The Fragility of Multi-Treebank Parsing Evaluation (2022)

Abstract: Treebank selection for parsing evaluation and the spurious effects that might arise from a biased choice have not been explored in detail. This paper studies how evaluating on a single subset of treebanks can lead to weak conclusions. First, we take a few contrasting parsers, and run them on subsets of treebanks proposed in previous work, whose use was justified (or not) on criteria such as typology or data scarcity. Second, we run a large-scale version of this experiment, create vast amounts of random subsets of treebanks, and compare on them many parsers whose scores are available. The results show substantial variability across subsets and that although establishing guidelines for good treebank selection is hard, some inadequate strategies can be easily avoided.

## introduction a limitation in nlp evaluation lies in the association between solving a dataset versus solving a task. datasets are domain-specific, their sizes differ and they are only available for a handful of languages and cultures @xcite . yet, we often ignore that the chances that these results generalize in the real world are scarce. in this context, the conclusions extracted from a single dataset should be taken with caution. for dependency parsing, the universal dependencies framework (ud; zeman et al., 2020) mitigates some of these issues. for instance, version 2.8 of ud includes 202 treebanks and 114 languages covering diverse linguistic typologies, treebanks with different amounts of data, and domains. paradoxically, this also complicates decisions when it comes to comparing dependency parsers in multilingual environments, which can be summarized as: how to choose a small but representative set of treebanks? although there are shared tasks @xcite that do consider experiments over a wide set of treebanks and help understand parsing models, such setups do not usually stick when the shared tasks end, and authors often run their models only in a handful of treebanks @xcite @xcite . this mostly happens for justified reasons: lack of computational resources to train the models in a reasonable amount of time, energy usage concerns, difficulties to summarize large experiments, or interest in specific phenomena (e.g. non-projectivity). thus, a good treebank selection strategy is crucial to reduce the chances of selecting an unrepresentative subset of treebanks, which could lead to weak conclusions. furthermore, even when using the whole ud collection is viable, treebank selection can still be relevant as ud is not a representative sample of languages (e.g., 62 out of the 114 languages in v2.8 are indo-european), so coarse-grained measures like averages over all treebanks may be misleading. ## related work the appropriateness of experimental setups for parsing evaluation has been studied in recent years from different perspectives. some authors have focused on determining what are the treebank particularities that make some of them easier to parse than others. for instance, the size of the training set is widely known to be an important factor to obtain accurate results in dependency parsing @xcite . other aspects such as domain similarity @xcite or annotation similarity @xcite between the training and test sets have also been studied, showing that they can greatly affect the performance of parsers. other particularities that can also affect the performance on a treebank are linguistic variation @xcite , annotation criteria @xcite , arc direction @xcite , average dependency length @xcite , non-projectivity @xcite , morphological richness @xcite or information-theoretic metrics @xcite , among other factors. although not specifically for parsing but nlp, @xcite and @xcite comment that the way data is split can play a role on test results, and thus on conclusions. extrapolating this to parsing, it would suggest that some parsers could obtain better results for certain treebanks just due to data splitting decisions, and not due to a linguistic motivation that would explain a given language being harder to parse than other. recently, søgaard (2020) studied the influence of overlap between trees in training and test sets in a given split, and concluded that (the amount of) graph isomorphism between the training and test set trees partially explains why some treebanks are easier or harder to parse than others. however, @xcite replicated the study, controlling for covariants, and proved that much of this observation is explained by relevant covariants like treebank size and mean test sentence length. another line of research more related to our work involves the studies that compare how different parsing algorithms behave on the same heldout test sets. @xcite showed that non-neural transition-based and graphbased parsers perform overall similarly, but produce different types of errors, with transition-based parsers being weaker for long dependencies and graph-based parsers weaker for shorter, more local ones. relatedly, de lhoneux et al. (2017a) compared a neural and non-neural transition based parser, showing that the former is not only clearly better at longer dependencies, but that it also needs less training data to parse effectively. @xcite replicated the work by mcdonald and nivre for neural versions of those parsers and, contrarily, demonstrated that the contextualization of the input vectors with recurrent networks results into both types of parsers showing a much more homogeneous behavior. also related to this, anderson and gómez-rodríguez (2020b) showed how different transition-based algorithms are prone to outperform others on a specific treebank according to their inherent dependency displacement biases. to the best of our knowledge, there have been only two papers in the literature that specifically focus on presenting methodologies to choose a suitable set of treebanks for parsing evaluation, both centered on ud and with the goal of obtaining a small sample of treebanks that is representative of the full ud collection (not necessarily of human languages as a whole). de lhoneux and nivre (2016); de @xcite do so by manually selecting treebanks to enforce typological diversity as well as representativity in other relevant aspects for parsing, like projectivity or treebank size. in turn, @xcite take an automatic, quantitative approach, obtaining a sample by clustering using delexicalized parsing performance. while many other papers have presented and used subsets of ud treebanks for evaluation, they either do not focus on representativity (e.g. @xcite ) or follow one of these methodologies (e.g. anderson and gómez-rodríguez (2020a)). ## hyphothesis and methodology as suggested above, parsing conclusions on multilingual environments are usually drawn from empirical research, which are prone to be parser-specific, experiment-specific, as well as treebank dependent. hypothesis we delve into this problem and hypothesize that parser comparisons based on running experiments and taking accuracy metrics on a given (reasonably-sized) subset of treebanks may lead to weak conclusions on rankings or differences in performance; as the magnitude and/or sign of the differences between parsers can change substantially depending on the choice of said subset. ## experiment 2: many parsers, many randomized subsets this experiment can be seen as a re-definition of experiment 1 at a large scale. above, we compared a few competitive parsers only on a handful of subsets of treebanks that were human-defined, and observed different trends. yet, this is a limited view of the problem. if we take as reference the @xcite and the 82 treebanks that were evaluated, considering subsets of size 10 (meaning that each is composed of 10 different treebanks), we would obtain up to ∼ 2.13 × 10 12 possible combinations. many of those subsets will not be a representative sample of languages, but we already saw that there are subsets that are used in parsing as a benchmark that are not either, and that even when they are considered representative, the criteria varies, and the parsing performance, differences among parsers and error reductions vary too. here, we generate subsets, similar in size to typical human-defined ones, and see how subset differences affect parsing rankings. similarly to experiment 1, we do not analyze here algorithms and parsing architectures, or their correctness, but the appropriateness of evaluation procedures. in this context, some shared-task systems have reported bugs in their pipeline: this was mostly evident for some systems that consistently ranked in the last positions (see ## discussion we have designed two experiments that revealed issues of relying on a single subset of treebanks for parsing evaluation. more particularly, we have shown that: (i) existing human-defined subsets show high variability in terms of rankings and performance across parsers, (ii) parsers that have been developed on a concrete subset might be biased towards performing better on that subset, (iii) it is relatively easy to come up with subsets that generate different parsing rankings, (iv) this can even happen across subsets that have been purposefully defined to be representative, (v) both linguistic typology and resource size have a large influence in the variability of results between parsers, and (vi) linguistic factors can be crucial even when parsers are designed in a language-agnostic way. overall, some advice can be given: (a) claims that "parser x is more accurate than parser y" can be weak even on carefully selected samples of ud treebanks (and perhaps it is recommendable to consider metrics that take into account dimensions 12 clas: it ignores selected relations which attach function words to content words. finally, there are aspects that we did not study in this piece of work, but that could affect the robustness of parsing evaluation as well, e.g., automatically versus manually annotated treebanks, and interactions between language and treebank properties (e.g. morphological complexity, dependency distance, . . . ) and parsing models. ## conclusion different subsets of treebanks have been proposed to try to capture the essence of the whole set of ud treebanks, so that the performance of parsers in such subsets would be representative of that obtained in the full set. we have empirically shown limitations of this approach, and also how establishing guidelines for good treebank selection can be hard, although some bad practices can be avoided.