Revisiting the Effects of Leakage on Dependency Parsing (2022)

Abstract: Recent work by Søgaard (2020) showed that, treebank size aside, overlap between training and test graphs (termed leakage ) explains more of the observed variation in dependency parsing performance than other explanations. In this work we revisit this claim, testing it on more models and languages. We find that it only holds for zero-shot cross-lingual settings. We then propose a more fine-grained measure of such leakage which, unlike the original measure, not only explains but also correlates with observed performance variation. Code and data are available here: https://github.com/miriamwanner/reu-nlp-project

## introduction syntactic parsing has long been one of the core natural language processing (nlp) tasks, and the proliferation of the universal dependencies project (ud; de marneffe et al., 2021; nivre et al., 2017) has allowed the development and comparison of monolingual and multilingual models under the same syntactic framework. the performance of the dependency parsers, however, varies wildly across languages, with stateof-the-art performance ranging from labeled attachment scores below 20 (e.g. for amharic, erzya, komi, or yoruba) to more than 90 (e.g. for spanish, polish, russian, or greek). as the ud treebanks follow mostly similar annotation guidelines, comparisons of the parsing performance across languages are now possible, to an extent. @xcite in an effort to explain these cross-lingual performance differences, researchers have proposed treebank size (vania et al., 2019), linguistic variation (nivre et al., 2007), test data sentence length miriamwanner/reu-nlp-project 2 different treebank creation protocols followed across languages (whose effects are hard to isolate or measure) can be a significant source of variation. nevertheless, some of the observed variation can be possibly explained by other factors. we direct the reader to footnote 2 of (søgaard, 2020). dependency trees: she saw it pron verb pron root nsubj dobj the big boat det adj noun root det amod søgaard (2020) unlabeled directed graphs: node-and edge-labeled directed graphs: rt v pr pr nsubj d o b j rt n dt a det a m o d figure 1 : only labeled reductions produce different graphs for these fundamentally different sentences. under unlabeled leakage, the two trees do leak. when taking labels into account the two trees belong to different isomorphisms and are not considered "leaky". or average gold dependency length (mcdonald and nivre, 2011), and domain differences between training and test data (foster et al., 2011), as potential predictors. recently, søgaard (2020) proposed that the proportion of isomorphic graph structures between the training and testing data (leakage) is a stronger predictor of the parsers' performance than any of the previously listed attributes other than training treebank size. søgaard (2020) concludes that "some languages seem easier to parse because their treebanks leak." this finding is potentially crucial for current parser evaluation on the existing treebanks, as well as for future treebank construction. it implies, for instance, that parsers are perhaps not as good as they seem, because they are tested on "leaky" test data. perhaps one should also consider designing treebanks that do not leak between train and test, as such a test set would not have a bias toward more common phenomena. in this work, we examine this finding more closely. we extend søgaard's definition to include labeled leakage, and study it over multiple parsers in both monolingual and cross-lingual settings. we show that the finding does not hold up when tested against more modern parsers and more languages. we do identify, though, that leakage indeed predicts parser performance in zero-shot cross-lingual settings, and we dive deeper in this phenomenon with an extensive study focusing on faroese and other germanic languages. last, we propose a modification of the leakage measure that both predicts and correlates with parser performance in such settings. ## leakage and how to measure it in this section we first define leakage based on graph isomorphisms and reproduce søgaard's experiments. we then show that parsers make local decisions that allow them to generalize to unseen graphs, and explore additional measures of leakage, studying whether they help explain parser performance. last, we argue that sub-trees are more meaningful units than label-free, tree-level representations. leakage definition leakage can be broadly defined as the portion of test trees that have isomorphic counterparts in the train set. while dependency trees are labeled, directed graphs with labels both on the nodes and on the edges, søgaard (2020) performed a reduction by removing labels from both nodes and edges. given these reduced graphs, søgaard (2020) finds the different isomorphisms that are present in the training and the test set, using the vf2 algorithm (cordella et al., 2001). we note that the isomorphism may or may not rely on node or edge labels. in the experiments below, we perform an ablation between using completely unlabeled directed graphs, node-labeled (but not edge-labeled) directed graphs, and between using the full information of the graphs to compute isomorphisms, namely both node and edge labels. reproducing (søgaard, 2020) examples of the reductions needed for computing leakage for two sentences are shown in figure 1 . now, assume that the first sentence is in the training set and the second is part of the test set. measuring leakage without labels implies that the first dependency tree is somehow informative for producing the tree for the second sentence, which we believe is counterintuitive. hence, our first hypothesis is that a more informed leakage calculation is going to explain more of the performance variance. we reproduce the experiments of søgaard (2020) comparing the three different reductions (denoted as "none" for unlabeled graphs, "edges" and "nodes+edges" for respectively labeled graphs). the experiment consists of correlating the factors φ assumed to influence syntactic dependency parser performance with the performance of the parser under study. we train a simple linear regression model foot_0 with treebank size and φ as input and parser performance as output. φ will correspond to our measure of treebank leakage. mathematically, we have αt s +βφ+γ with t s treebank size and α, β, γ learned parameters. following søgaard, we will focus on explained variance and mean absolute error (mae) from five-fold cross-validation to avoid overfitting. unlike søgaard, we will additionally report spearman's ρ correlation coefficients foot_1 between factor and performance, which will reveal whether indeed leakage leads to better parser performance. @xcite the results on the same data as søgaard (2020) (using the best reported parser performance from the conll 2018 shared task) are presented in the top three rows of while søgaard (2020) was correct (for the languages/parsers they studied) to state that there is a correlation between leakage and parser performance, we believe they reached an incorrect conclusion. the metric they used (explained variance) does not reveal the direction of the correlation, just that there is a correlation. because of this they came to the wrong conclusion that there was a positive correlation between leakage and parser performance. our results ( ## leakage explains 0-shot performance modern dependency parsing models trained on many languages perform well on languages unseen (zero-shot setting) during training (muller et al., 2021; glavaš and vulić, 2021, inter alia). we focus again on udify, since it performs well in zero-shot settings. this is generally attributed to two factors: the presence of related languages in the training set, and the multilingual capabilities of the underlying representation model (here, a multilingual bert (devlin et al., 2019) model). analysis on faroese looking deeper into the zero-shot setting, we perform an experiment on a simplified bilingual zero-shot setting. we train parsers in five germanic languages (german, swedish, danish, norwegian, icelandic) and test on faroese in a zero-shot fashion. for each language, we train a model on: for example, take german training and faroese test sets. first all german instances leaking into faroese are added to the "german leaky" train set and the corresponding leaked faroese sentences are put into the "faroese-leaky" test set. then the remaining sentences from the german training set are added to the "german nonleaky" train set and the remaining sentences from the faroese test set are the "faroese nonleaky" test set. last, we take a random sample of same size across all settings, so that training data size is not a confounding factor for our analysis. see figure 3 and extensive results in appendix c. for all languages, models trained on leaky data perform worse than models trained on the same amount of non-leaky or random data. for most transfer languages, in fact, training solely on nonleaky data performs better than training on other subsets! in addition, the leaky part of the testing data is clearly easier to parse in general, while the non-leaky part is more challenging. the models trained on perfectly 'diverse' treebanks generally perform just as good as those trained on non-leaky or randomly sampled data and often better on the non-leaky test set, which means they generalize better. this indicates a way to reach better cross-lingual performance without the need for large training data, as long as the training set is diverse enough. the large performance difference between models trained on leaky and non-leaky trees reveals that something is different about the parts of the treebanks that leak. we measured the diversity of the leaky, non-leaky, and randomly selected trees, defined as the number of unique trees divided by the total number of trees. we found that leaky treebanks were far less diverse and therefore contain fewer unique structures than non-leaky or randomly sampled counterparts. across all treebanks, leaky instances are also generally shorter (8.4 vs 21.6 avg length), shallower (2.2 vs 4.8 average tree depth), and with shorter avg dependency length (2 vs 3.3). we argue that the reasoning should be reverse: short "easy" examples are more likely to leak; it is not leakage that makes them easy! ## conclusion we re-evaluate claims that training-test leakage can explain parser performance, define a subtree-based leakage measure that better explains performance, and show that this claim only holds for zero-shot transfer settings.