Investigating UD Treebanks via Dataset Difficulty Measures (2023)

Abstract: Treebanks annotated with Universal Dependencies (UD) are currently available for over 100 languages and are widely utilized by the community. However, their inherent characteristics are hard to measure and are only partially reflected in parser evaluations via accuracy metrics like LAS. In this study, we analyze a large subset of the UD treebanks using three recently proposed accuracy-free dataset analysis methods: dataset cartography, \mathcal{V} -information, and minimum description length. Each method provides insights about UD treebanks that would remain undetected if only LAS was considered. Specifically, we identify a number of treebanks that, despite yielding high LAS, contain very little information that is usable by a parser to surpass what can be achieved by simple heuristics. Furthermore, we make note of several treebanks that score consistently low across numerous metrics, indicating a high degree of noise or annotation inconsistency present therein.

## introduction datasets have long played a crucial role in dictating the pace of progress in nlp. their function, for most tasks, is largely two-fold: 1) to collect data points (and their corresponding gold-standard labels) on which statistical models can be trained, and 2) to serve as benchmarks though which various models can be evaluated and compared. in recent years, much research has been devoted towards developing new datasets, tasks, and benchmarks for nlp -so as to articulate the distinguishing aspects of a bevy of new neural models. syntactic parsing has remained an active area of research in this regard, and universal dependencies (ud) @xcite @xcite has emerged as a crucial initiative within nlp, offering a set of cross-lingually consistent annotation principles that have since been adapted to over 100 languages. notably, the conll shared tasks of @xcite @xcite featured ud at the forefront, inviting researchers to submit systems that could not only parse, but process the entirety of ud across its numerous annotation layers. beyond parsing, ud has also been utilized for a variety of other ends, including cross-lingual transfer @xcite @xcite , domain adaptation @xcite @xcite , and linguistic typology @xcite @xcite . though ud and other initiatives have aided in driving recent advances in nlp, overall progress has typically been measured via aggregate accuracy metrics, which provide little more than a bird's eye view into the data. in the era of deep learning, where popular models are notoriously opaque, it has thus proven vital to study the contents of datasets and identify aspects that may misrepresent model performance. in this vein, numerous studies have shown that the crowd-funded nature of some popular nlp datasets makes them prone to annotation artefacts that are readily exploitable by neural models as heuristics @xcite @xcite . with such insights in mind, researchers have shifted their focus towards the datasets instead of the models, proposing general methods for exploring the former so as to better understand the performance of the latter. such approaches have drawn from, e.g., information theory @xcite , item response theory @xcite , and model training dynamics @xcite . this work, however, has predominately focused on classification tasks and has proven difficult to extend to other classes of problems, such as the structured prediction tasks of ud. in this paper, we perform an analysis of (a large subset of) ud v2.9 through the perspective of a popular parsing architecture -namely that of @xcite . as opposed to much previous work, which prioritizes metrics like las in order to build accurate parsers, we aim instead to better understand the underlying data, as well as how our parser interfaces with it. to do so, we extend recently proposed dataset analysis methods based on model training dynamics @xcite , v-information @xcite , and minimum description length @xcite @xcite to the dependency parsing scenario. in working with each method, we formalize the following set of research questions: ## beyond accuracy in nlp, system performance is typically measured via accuracy, which has the advantage of being intuitive and straightforward to calculate. more-over, it is often useful in model selection, as well in setting the state-of-the-art for a given dataset or task. beyond these qualities, however, accuracy leaves much to be desired when investigating models vis-a-vis datasets. for one, it is often reported with respect to a single checkpoint from a model's training regime, which typically consists of numerous epochs and parameter updates. in honing in on one particular checkpoint (usually the best with respect to validation loss or accuracy), one cannot readily assess whether the model was easily fit on the data, or if training was stopped prematurely. furthermore, in choosing the arg max over the output distribution, one inevitably loses information about it: was the model confident in making its prediction? or was the distribution highly entropic? also relevant is the train/test distinction: in evaluating on the latter, one can gauge a model's ability to generalize, but generally cannot assess the goodness-of-fit on the former, nor its sample efficiency. likewise, accuracy cannot, in principle, adequately assess the quality of the training data: can the model learn from all instances therein? or does the data contain a substantial amount of noise due to, e.g. annotation inconsistencies? with regard to dependency parsing, accuracybased metrics like las carry a number of additional drawbacks. for example, if working with an arc-factored graph-based dependency parser, one must score all possible n incoming edges for a dependent x @xmath0. if @xmath1, then x must necessarily be attached to the dummy node and assigned the root label. for a treebank consisting of many such sentences (e.g. russian taiga), this will lead to artificially inflated accuracies for that particular relation. another concern is the potential proliferation of functional relations that may arise in some treebanks (e.g. japanese gsd), where parsers often yield disproportionately high accuracies @xcite . to transcend the limitations of accuracy-based measures in dependency parsing, we consider three recently proposed dataset analysis methods as a means of exploring ud treebanks: dataset cartography @xcite , v-information @xcite , and minimum description length @xcite . ## experimental setup in this section, we describe our data sampling procedure, the parsing model we employ, and our extension of the aforementioned analysis methods to the context of (graph-based) dependency parsing. more details about how each metric is calculated can be found in appendix a. ## results and analysis in this section, we analyze the results of our four metrics: conf, var, v-info, mdl. we focus our analysis on arcs (probabilities and loss) and refer to appendix b for the full set of results. for each metric, we begin with a brief discussion of its pairwise correlation to all other metrics, which is displayed for reference in figure 1 . we continue by framing our analyses with respect to the top 3 best and worst performing treebanks for each metric. ## conclusion in this paper, we investigated 88 ud treebanks through the lens of dataset difficulty measures. we found that conf and var are capable of painting a nuanced picture of how easy or hard treebanks might be to parse. we also observed that a model's confidence throughout training is an excellent indicator of how well it might generalize to held-out data. regarding v-info, we observed that the measure tends to simultaneously penalize high degrees of predictability and stochasticity, and that treebanks otherwise characterized as easy may have low v-info due to lack of structural diversity. finally, treebanks with high mdl seem to be characterized by low sample efficiency, which in turn is related to morphological complexity and vocabulary usage. given the broad range of insights expressed via these metrics, we hope that our results -however preliminary -will inspire future researchers to pursue a greater understanding of ud as trove of data, so as to push further boundaries in the realms of parsing, typology, etc. in terms of future work, we make note of several potentially interesting directions. indeed, one of the main drawbacks of our experimental design is that it only accounts for the perspective of a single parsing architecture -albeit (arguably) the most popular one of the neural era. dependency parsing, however, has a long research tradition where many different parsing models have been proposed throughout the years -each with their respective advantages and drawbacks (see, e.g. @xcite ). though we chose to work with an arc-factored graph-based parser due to the need for extracting arc-level probabilities, future studies may consider consider ways of leveraging transition-based parsers @xcite or models that directly maximize full tree probabilities @xcite . if working with a wide range of parsing models, one could employ item response theory @xcite , which is a framework that consolidates many predictions per instance in order to identify regions of datasets that may be perceived as difficult, easy, etc. certainly, this would provide a more broad perspective on ud than what we have offered here. a different direction that could be explored is data selection, which is indeed what @xcite proposed as the main uses for dataset cartography. although the conf and var metrics provide valuable insights about ud treebanks in our case, they are nonetheless measured at the token-level. this is distinct from their original application, in that each token is crucial for the composition a sentence and cannot be readily removed. although we experimented with measuring sentence-level conf and var, some preliminary results indicated that a naive application of @xcite 's method is ultimately confounded by sentence length. as such, it would be interesting to experiment with models that directly optimize for tree probability, such as @xcite . if successful, this would allow us to identify select subsets of treebanks for the purpose of training more accurate parsers with less data, or for choosing the least noisy sentences for typological studies, etc.