Towards Instance-Level Parser Selection for Cross-Lingual Transfer of Dependency Parsers (2020)

Abstract: Current methods of cross-lingual parser transfer focus on predicting the best parser for a low-resource target language globally, that is, “at treebank level”. In this work, we propose and argue for a novel cross-lingual transfer paradigm: instance-level parser selection (ILPS), and present a proof-of-concept study focused on instance-level selection in the framework of delexicalized parser transfer. Our work is motivated by an empirical observation that different source parsers are the best choice for different Universal POS-sequences (i.e., UPOS sentences) in the target language. We then propose to predict the best parser at the instance level. To this end, we train a supervised regression model, based on the Transformer architecture, to predict parser accuracies for individual POS-sequences. We compare ILPS against two strong single-best parser selection baselines (SBPS): (1) a model that compares POS n-gram distributions between the source and target languages (KL) and (2) a model that selects the source based on the similarity between manually created language vectors encoding syntactic properties of languages (L2V). The results from our extensive evaluation, coupling 42 source parsers and 20 diverse low-resource test languages, show that ILPS outperforms KL and L2V on 13/20 and 14/20 test languages, respectively. Further, we show that by predicting the best parser “at treebank level” (SBPS), using the aggregation of predictions from our instance-level model, we outperform the same baselines on 17/20 and 16/20 test languages.

## introduction despite recent evidence that, for pretrained transformers like bert @xcite , explicit syntax contributes negligibly to better natural language understanding @xcite , proper language-specific automated syntactic analyses are still paramount in numerous tasks in computational linguistics, yet unavailable for 99% of the world's languages due to the lack of respective treebanks. a major goal and promise of cross-lingual transfer in nlp is to transfer language technology to as many languages as possible @xcite @xcite . therefore, cross-lingual transfer of dependency parsers has profiled as the most viable strategy to use parsing technology in resource-low languages @xcite @xcite . delexicalized transfer is conceptually the least demanding option in terms of language-specific resource requirements. the only provision, in order to transfer the parser trained on a delexicalized treebank of a resource-rich language, is a pos tagger in a low-resource target language based on the universal pos (upos) tagset @xcite . delexicalized transfer is nowadays used primarily as a simple yet competitive baseline for more sophisticated transfer models when porting parsing technology in a new language. however, in realistic truly low-resource setups, one cannot guarantee additional resources such as parallel sentences @xcite @xcite @xcite , word alignments @xcite , sufficiently large monolingual corpus in the target language @xcite , and language coverage in massively multilingual language models which are used as the basis for modern parsers @xcite . thus, delexicalized transfer still remains a widely useful baseline and plausible option @xcite . cross-lingual transfer comes in two main flavors. we either (1) choose the best parser from a set of available parsers, trained on treebanks of various resource-rich languages (single-best parser selection, sbps) or (2) use the parser trained on a mixture of treebanks of (ideally related) resource-rich languages (multi-source parser transfer, msp). other transfer paradigms, like data augmentation @xcite , assume the existence of at least a small treebank for a target language, violating the assumption of a (treebank-wise) fully low-resource target language. both sbps and msp rely on some measure of structural alignment between languages in order to select either the single best source language parser (sbps) or a set of (syntactically related) source languages (msp). existing solutions rely on measures like the kullback-leibler (kl) divergence between sourceand target-language distributions of pos trigrams @xcite , which can be unreliable for small target language corpora or instance-level estimation. more recent approaches @xcite choose suitable source languages based on manually coded typological similarities between languages available from databases such as wals @xcite or uriel @xcite . the bottleneck of this approach is the manual effort and linguistic expertise needed to introduce a new language into the database. ## motivation: the case for instance-based parser selection the idea behind instance-level parser selection is intuitive: given a set of parsers for resource-rich source languages, it is unlikely that the same source-language parser is the best choice for all instances (i.e., upos-sentences) of the target language. therefore, we first investigate the performance of an oracle model that would be able to predict the best source-language parser for each individual pos-sentence from the target-language treebank. to verify this, we rely on the well-known biaffine parser @xcite and train it on delexicalized ud2.3 treebanks @xcite of 42 languages. 2 we then parse the delexicalized treebanks of the 20 low-resource languages with all 42 source parsers, and measure their performance per each instance in each target treebank. we compare the performance of two oracle parser selection strategies: (1) single-best parser selection (sbps), in which for each target test treebank we select the parser that performs best on the entire treebank; and (2) instance-level parser selection strategy (ilps), where for each upos-sentence from each test treebank, we select the parser that produces the best parse for that upos-sentence. the differences in unlabeled attachment scores (uas) between the two transfer paradigms are shown in figure 1 . this clearly demonstrates a large gap in favor of ilps: the average gain with ilps is 14.5 uas points, and it is prominent for all languages. it suggests that large improvements may be obtained with a model that can predict the best parser at the instance level, that is, for each upos-sentence separately. however, these are still oracle scores and we pose the following research questions in this paper: (q1) is it possible to learn an instance-level prediction model to select the best parser given any upos-sentence, irrespective to its "language of origin"? foot_2 in addition, even with noisy automatic instance-level predictions, one could still, by eliminating the noise through aggregation, use them to inform treebank-level source parser selection. in other words, another research question we pose is: (q2) can we improve single-best global parser selection through aggregating instance-level parser predictions? ## instance-based parser selection we now describe a novel ilps framework based on a supervised regression model that predicts the parser accuracy for any upos-sentence. as such, it can be applied on upos-sequences of low-resource languages. as described in §2, we first train a (biaffine) parser on delexicalized treebanks for each of the 42 resource-rich languages from ud2.3. 4 we then parse with each parser the 41 treebanks of the other languages. this way we obtain the labels for training the ilps regression model. the data preparation step is further detailed in §3.1, while the regression model itself is described in §3.2. at inference time, the ilps model predicts the accuracy of each of the 42 parsers for each upos-sentence from delexicalized treebanks of the 20 test languages. note that this constitutes a minimal-resource and true zero-shot language transfer setup: our ilps regression model does not rely on any information about the test languages nor their respective treebanks. finally, in §3.3 and §3.4, we outline different strategies for merging the parse trees based on the predictions of the ilps regression model. the full ilps framework is illustrated in figure 2 . ## experimental setup data. we perform all experiments on the ud v2.3 dataset, foot_6 as it contains a wide array of both resourcerich languages with large treebanks -split into train, development, and test portions -and low-resource languages with small test treebanks. for our experiments, we select 42 languages with the largest treebanks as our resource-rich source languages for training, and a set of 20 typologically diverse low-resource languages for testing. 8 following established practice @xcite , at inference we use gold upos-tags of test treebanks for all models in comparison. 9 ilps hyperparameters are optimized via fixed-split cross-validation on our training set (see §3.1). we set the embedding size for both parser embeddings and upos-tag embeddings, as well as the hidden size of the feed-forward transformer layers to 256. the transformer encoder has n @xmath0. we update the model in mini-batches of 16 examples, using adam (kingma and ba, 2015) with the default parameters: β 1 = 0.9, β 2 = 0.999, @xmath1, with an initial learning rate set to 10 -4 . the regression mlp has 2 hidden layers with 256 units each, plus a linear projection layer that compresses the 256-dimensional vector into a single prediction score. we perform early stopping based on the loss on the development set. for all ensembles, we set the parser inclusion threshold τ to 0.9. oracle scores and baselines. in order to provide more context for the reported ilps scores, we also report the results of two oracle methods described in §2: the oracle single-best parser selection (or-sbps), and the oracle instance-level best parser selection (or-ilps). we compare to three competitive baselines: (1) the standard multi-source parser (msp) baseline which trains a single parser model on the concatenation of all training treebanks; 10 and two competitive sbps baselines, (2) kl-sbpstreebank-level parser selection based on the kullback-leibler divergence between upos-tag trigram distributions of the source and target language treebanks @xcite and (3) l2v-sbps -treebank-level parser selection based on the cosine similarity between the syntax-based vectors of the source and target language from wals @xcite . ensembles. we evaluate two ensembles based on the predictions of our ilps-based regression model, described in §3.3: (1) an instance-level ensemble in which we merge the trees of the best parsers for each sentence (ens-ilps) and ( 2 ) ens-sbps ilps -an ensemble merging the trees of treebank-level best parsers, where the treebank-level estimates are aggregated from the instance-level predictions. we evaluate comparable ensembles (i.e., with the same parser inclusion performance threshold τ = 0.9) for both sbps baselines: ens-kl-sbps and ens-l2v-sbps. ## results and discussion we first show the results for single-best parser selection models. we then proceed to a more realistic ensemble setup in which the models are allowed to select more than just one parser. single-parser selection. we report results (uas) for all single-parser selection methods (i.e., no ensembles) along with the oracle scores on all 20 test treebanks. ilps and sbps ilps still do not match the performance of the multi-source parser (msp) in this simple single-parser-selection setup. we find this somewhat expected: ilps and sbps ilps are based on parsers trained on single treebanks, whereas msp is trained on the concatenation of all training treebanks. therefore, we include msp as a baseline in our ensemble evaluation as well. ensemble evaluation results. we show the results for the ensemble models in ## related work parsing languages with no training data has been a very active topic of research for nearly a decade since the pivotal works by @xcite . many diverse approaches are explored along the lines of model transfer, annotation projection, machine translation @xcite @xcite @xcite , and selective sharing based on language typology @xcite and structural similarity @xcite . however, vast majority of prior work involves bulk evaluation, whereby transfer parsers are validated by mean accuracy on test data. such evaluation protocols stand in contrast with the fact that languages exhibit high variance in syntactic structure, which calls for a sensitive treatment of every sentence. while an oracle single-source parser may be appropriate for the majority of sentences in a given dataset, instance-based treatment closes the gap to the best achievable result given an array of pretrained parsers, as we also show in §2. early efforts in this line of research include data point selection where language models are used to capture the prevalent syntactic structure of a language and score potential training instances such that a multi-source parser is trained on the mixture of training instances that are most similar to the test language instances @xcite . instead of instance selection one can also apply instance reweighting in accordance to their similarity to the test language @xcite . regardless of whether we attempt to align languages on an instance-level or on a treebank-level there is a need for a similarity measure between languages. prior work relied on existing manually curated resources such as the uriel database @xcite , using the kl-divergence on pos-trigrams @xcite , or handcrafted features derived from the datasets at hand. our work is most similar to the recent work of @xcite : they learn to score and rank languages in order to predict the top transfer languages. however, contrary to their work, our approach does not employ a model to learn the ranking, but transforms the labels to directly reflect the ranking when we train the scoring model. in addition, we stress the importance of instance-based learning for cross-lingual parser transfer in particular. another core difference is that our approach is an end-to-end system without external resources or handcrafted static features. instead, our framework relies on trainable parser embeddings that encode the necessary features in a single representation. from another viewpoint, the work of @xcite @xcite explores the potential of synthesizing and reordering delexicalized pos sequences to come up with better parser transfer without unrealistic assumptions on target-language resources. their work in synthetic delexicalization is compatible with ours as it lends itself entirely to instance-based parsing. finally, the line of work by @xcite in learning monolithic models over multiple languages, and its continuation for zero-shot learning by @xcite also promises to abstract away from language boundaries, but still records significantly lower zero-shot scores than our proposal. ## conclusion and future work in this work, we indicated that there is a large disparity between mean test-set and per-instance accuracy in cross-lingual parser transfer setups. we showed convincing evidence that one source parser is not the optimal choice for all target-language sentences. motivated by the analysis, we proposed a novel approach to close this gap in this proof-of-concepty study: instance-based parser selection. our framework provides competitive results, where in the ensemble setting we outperform all baselines, and markedly even the single-source oracle parser selection, while using a simple thresholding heuristic to select the parsers. we see the proposed model as the first exploratory step in the direction of robust instance-level parser transfer, which opens several avenues for future research. while this proof-of-concept work assumed the existence of gold pos tags, we will also experiment with the same approach "in the wild", with learned or transferred pos taggers, and we will also extend the study to lexicalized parser transfer following the latest developments in the domain of lexicalized multilingual and cross-lingual parsing @xcite . future work may also include learning to rank parsers instead of applying simple heuristics. further improvements may be obtained by using the accuracy predictions from our model as a feature and combining it with external linguistic features @xcite . the idea of instance-based parser selection lends itself also to domain adaptation settings, following @xcite , even for well-resourced languages.