Are UD Treebanks Getting More Consistent? A Report Card for E nglish UD (2023)

Abstract: Recent efforts to consolidate guidelines and treebanks in the Universal Dependencies project raise the expectation that joint training and dataset comparison is increasingly possible for high-resource languages such as English, which have multiple corpora. Focusing on the two largest UD English treebanks, we examine progress in data consolidation and answer several questions: Are UD English treebanks becoming more internally consistent? Are they becoming more like each other and to what extent? Is joint training a good idea, and if so, since which UD version? Our results indicate that while consolidation has made progress, joint models may still suffer from inconsistencies, which hamper their ability to leverage a larger pool of training data.

## introduction the universal dependencies project 1 @xcite has grown over the past few years to encompass not only over 100 languages, but also over 200 treebanks, meaning several languages now have multiple treebanks with rich morphosyntactic and other annotations. multiple treebanks are especially common for high resource languages such as english, which currently has data in 9 different repositories, totaling over 762,000 tokens (as of ud v2.11). while this abundance of resources is of course positive, it opens questions about consistency across multiple ud treebanks of the same language, with both theoretical questions about annotation guidelines, and practical ones about the value of joint training on multiple datasets for parsing and other nlp applications. in this paper we focus on the two largest ud treebanks of english: the english web treebank (ewt, @xcite ) and the georgetown university multilayer corpus (gum, zeldes 2017). 2 al-though both datasets are meant to follow ud guidelines, their origins are very different: ewt was converted to ud from an older constituent treebank @xcite into stanford dependencies @xcite and then into ud, while gum was natively annotated in stanford dependencies until 2018, then converted to ud @xcite , with more material added subsequently via native ud annotation. coupled with gradual changes and clarifications to the guidelines, there are reasons to expect systematic dataset differences, which ud maintainers (including the authors) have sought to consolidate from ud version to version. despite potential pitfalls, nlp tools are increasingly merging ud datasets for joint training: for example, stanford's popular stanza toolkit @xcite defaults to using a model called combined for english tagging and parsing, which is trained on ewt and gum (including the reddit subset of gum). 3 we therefore consider it timely to ask whether even the largest, most actively developed ud treebanks for english are actually compatible; if not, to what extent, and are they inching closer together or drifting apart from version to version? regardless of the answer to these questions, is it a good idea to train jointly on ewt and gum, and if so, given constant revisions to the data, since what ud version? ## related work much previous work on consistency in ud has focused on cross-linguistic comparison, and especially on finding likely errors. some papers have taken a 'breadth-first' automatic approach to identifying any inconsistencies @xcite , with the caveat that many types of differences are hard to detect. others have taken a more focused approach to particular phenomena, fewer studies have examined cross-corpus parsing accuracy (alonso and zeman 2016 for spanish on ud v1.3, @xcite for russian using ud v2.2), and fewer still have looked at parsing consistency and stability @xcite . however to the best of our knowledge, no previous study has examined changes in consistency across ud versions, i.e. whether crosstreebank compatibility is increasing over time, how much so, and for which annotations? to see how data in both corpora has changed across versions, we use the official conll 2018 ud shared task @xcite scorer and compare each of the past six versions of each corpus to its next version, taking the updated version as an improved 'gold' standard. 4 this results in a score for each ud metric, such as the labeled attachment score (las), universal pos (upos) and englishspecific pos (xpos), as well as lemmatization and tokenization. figure 1 shows the difference 4 earlier comparisons are impossible since they predate gum's conversion to ud. in score between each pair of versions for each dataset, which we discuss for each corpus below. for example, taking v2.7 of ewt as the correction for v2.6, we see a 2% rate of tokenization errors (in green), indicating a substantial change in tokenization, but less than 0.2% change to v2.8, and zero changes to tokenization moving to v2.9. one caveat to note when working with data across versions is that unlike ewt, gum's contents are not frozen: the corpus grows with new material every year. in the overview below, we therefore keep the evaluation fixed and limited only to documents that have existed since v2.6 (136 documents, 120k tokens). in §4 we will consider scenarios using both this fixed subset and the entire corpus (193 documents, 180k tokens in v2.11). foot_3 ## cross-corpus parsing cross-corpus results to test whether ewt and gum are becoming more internally and mutually consistent, we train parsers on each version of each corpus, and test them against both corpora. if each corpus is becoming more consistent, we expect higher scores in each version; and if cross-corpus model scores are increasing, we infer that the data is becoming more consistent across corpora. to ensure a fair comparison, we keep training and test data from gum fixed to those documents that have been available since v2.6. however, since the experiments are limited to the smaller subset of ud gum v2.6 documents, they do not reflect current nlp tools (which train on all documents in the current ud repos), nor do they tell us whether joint training is a good idea. ## discussion in this paper we surveyed progress in consolidating the largest ud english corpora, ewt and gum. results show data is moving closer together: singlecorpus training still beats joint training by a hair, but joint models are nearly as good, and likely much more robust. as consolidation continues, we hope to see joint models overtake in-domain training, and more consistency expanding to other english datasets and other ud languages.