Identifying and Handling Cross-Treebank Inconsistencies in UD : A Pilot Study (2020)

Abstract: The Universal Dependencies treebanks are a still-growing collection of treebanks for a wide range of languages, all annotated with a common inventory of dependency relations. Yet, the usages of the relations can be categorically different even for treebanks of the same language. We present a pilot study on identifying such inconsistencies in a language-independent way and conduct an experiment which illustrates that a proper handling of inconsistencies can improve parsing performance by several percentage points.

## introduction merging several available treebanks for the same language is a simple way of obtaining more dataeither for training language-specific tools or for systematic corpus studies on the language. this strategy presupposes that each of the merged treebanks adheres to the same annotation scheme; when treebanks to be merged originate from distinct research contexts (which is typically the case in practice), a harmonization process is required. the ud initiative (whose main motivation has been to obtain comparable syntactic annotations across languages) has the welcome side effect that whenever several corpora for the same language are included in the ud collection, the amount of consistently annotated treebank data for this language increases -at least in theory. in practice, there can be serious inconsistencies in the way the dependency relations are used in the respective treebanks. this can for instance be the effect of distinct strategies in conversion from a non-ud treebank format, or of idiosyncrasies in the linguistic treatment of a certain construction pursued in one treebanking team. often such inconsistencies may go unnoticed -in particular when researchers do experiments on large samples of languages from ud -and may lead to problematic conclusions. some previous efforts have focused on discovering errors or inconsistencies within a treebank, by measuring the patterns based on partially lexicalized dependency relations @xcite or calculating the fitness scores of the dependency relations in a tree @xcite . in contrast to these studies, we focus on finding inconsistencies between treebanks that are supposed to be annotated in the same way according to a common guideline. our method builds on the assumption that each individual treebank is already largely consistent within itself, and seeks to identify dependency patterns with contradicting statistics across treebanks. so far, annotation inconsistencies have been reported for some specific languages, e.g. korean @xcite and russian @xcite , but there seems to be no comprehensive study addressing these cross-treebank inconsistencies within the ud collection. in our pilot study, we experiment with a dependency-based measure for inconsistency detection that makes no specific language-typological assumptions and appears to be quite effective for detecting various types of inconsistencies. in two experiments, we demonstrate that inconsistencies detected this way can be used as the trigger for a comparatively simple, but effective conversion process on one of the treebanks: a parser trained on the converted treebank achieves higher accuracy on the treebank that is closer to the ud guidelines. ## methodology we became aware of inconsistent usages of dependency relations when extracting relative branching directions for another task @xcite and we reuse this measure here to find potential cross-treebank inconsistencies, without making any prior assumptions what could be a problematic construction in a language under consideration. the measure takes advantage of the fact that the frequency distribution of the directionality of dependencies is by and large independent of domainspecific factors (which will of course differ across treebanks). moreover, many of the problematic inconsistencies found across treebank schemes go along with the choice of head vs. dependency status and will thus be reflected in the directionality of the arcs for a particular label -even when the label is used fairly infrequently. the relative frequency of right-branching instances for a dependency relation r in a treebank t is here, an expression of the form #[ * ](t) returns how often the construction * appears in the treebank t. the value ranges between 0 and 1; for example, v t @xmath0.e. always precede) their governor in t, 1 indicates that they always follow their governor in t, and 0.5 indicates that both orderings are equally common. this measure is very similar to the directional dependency distance of chen and gerdes (2017). our measure only quantifies the average direction of a relation whereas their measure quantifies average direction and distance. we found that the relative branching direction of a relation sometimes greatly differs among the treebanks of the same language. therefore, we calculate the maximum branching direction difference (mbd) for a language ℓ as mbd(r, ℓ) = max ## inconsistencies among treebanks we use the mbd to search for inconsistent usages of dependency relations in ud, version 2.5 @xcite . we show three examples in the following, two real inconsistencies and one case in which a higher mbd is caused by both consistent and inconsistent use of a relation. the rest of our analysis is provided in the appendix. we are not familiar with all languages of which we analysed treebanks and it is possible that we missed something, making our analysis possibly incomplete. this is also the reason for why we could not analyse the potential inconsistencies for some of the languages (marked by a question mark in the table). inconsistent chinese classifiers (clf) figure 1 shows that the annotation for classifiers is categorically different among the chinese treebanks. the treebanks cfl and hk follow the ud guidelines foot_3 and attach the classifier to the numeral whereas the other treebanks gsd and pud attach it to the noun. note that the annotation of numerals is also different; however, since the branching direction stays the same this cannot be detected by our method. inconsistent korean auxiliaries (aux) all three korean treebanks have a different usage of the "aux" relation (figure 2 ), which is already documented by @xcite : kaist is most closely to the ud guidelines, using the "aux" relation together with the "aux" tag in around half of its instances (but with a lot of different tags such as "noun" and "sconj" in the other half), whereas gsd generally uses "flat" and "verb" to connect auxiliary verbs with their main verbs. in both treebanks, "aux" connects main and auxiliary verbs from left to right. pud on the other side uses "aux" to connect main and auxiliary verbs from right to left and tags both with "verb". in-/consistent english compounds (compound) the use of "compound" is only partially inconsistent in english. "compound" connects common-noun compounds right-to-left, whereas "compound:prt" connects verbs and their particles left-to-right, and the partut contains a greater proportion of "compound:prt" instances than the other treebanks. this difference between the treebanks is detected by our method but not an inconsistency. however, the difference is additionally increased by many proper-name compounds in partut that are connected left-to-right with "compound". in the other english treebanks, similar compounds usually show a "flat" relation. 4 experiments zh_hk zh_pud zh_gsd.orig 48.14 54.97 zh_gsd.clf 50.25 51.51 (a) traditional chinese. zh_cfl zh_gsdsimp.orig 43.36 zh_gsdsimp.clf 46.78 (b) simplified chinese. ko_pud ko_kaist ko_gsd.orig 19.10 26.91 ko_gsd.aux 20.13 30.08 (c) korean. parsing performance of a model trained on one treebank of a language and tested on another treebank of the same language is generally affected negatively by the domain difference, but it will suffer even more seriously in case some of the arc labels are used inconsistently across the treebanks (even though the labels are taken from the fixed ud label set). to empirically demonstrate the impact of annotation inconsistency and verify that our method could alleviate the problem, we perform a pilot experiment on classifiers in chinese and auxiliaries in korean, as they are the top two in we designed a set of rules to convert the inconsistent relations in the training treebanks towards the ud guidelines foot_4 . the target test sets are left as they are, including the ones that are against the general ud guideline (the pud treebanks for both languages), since we cannot guarantee the correctness of the automatic conversion. for chinese classifiers and korean auxiliaries, there are almost perfect one-to-one correspondences between the annotation schemes which allows a simple conversion of zh_gsd/gsdsimp and ko_gsd, which is shown in figure 4 . we use the udpipe parser @xcite in our experiments with the default settings. it is based on the arc-standard projective transition system using a neural network classifier. it uses predicted pos tags and morphological features by the tagger. we then break down the parser accuracy on each individual relation. as expected, the f1 score of "clf" in the chinese treebanks improves from 0 to 69.96 for hk, from 0 to 84.34 for cfl, and from 65.05 to 0 for pud (since pud is annotated the same as the original gsd). for korean, the f1 score of "aux" changes from 0.93 to 63.53 for kaist, but remains 0 for pud, since pud annotates "aux" different from both gsd and kaist, as explained in section 3. generally, it is clear that by identifying and converting the inconsistencies, the parser model can learn to handle the relations consistent to the target treebank. it improves the parsing accuracy on the particular relations as well as the overall performance. ## conclusion and future work in this pilot study, we show that the measure of relative branching direction applied on two ud treebanks for the same language can be used as a simple, but effective means for detecting candidates for inconsistencies in the usage of ud relation labels -independent of prior assumptions of which relations are particularly problematic. we manually inspected a list of relations with high divergence of branching direction, and as a pilot experiment converted some identified relations and analyze the performance of a parser in the out-of-domain scenario. the experiment confirms the benefit of our identification and conversion of the inconsistent annotation. concurrently with this work, @xcite propose another method that addresses the identification of cross-treebank inconsistencies by measuring the differences of pos-tag trigrams. however, it focuses on a different type of inconsistency than what our method can identify. both types of inconsistency can occur independently and a joined approach of the two complementary methods could make further contribution to error detection/correction in the ud treebanks. pron verb noun pron pron verb it makes sense what she said expl obj csubj obj nsubj pron pron verb verb noun what she said makes sense obj nsubj csubj obj figure 6 : english annotation scheme for clausal subjects with expletive (above) and without expletive (below).