Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing (2022)

Abstract: We present substructure distribution projection (SubDP), a technique that projects a distribution over structures in one domain to another, by projecting substructure distributions separately. Models for the target domain can then be trained, using the projected distributions as soft silver labels. We evaluate SubDP on zero shot cross-lingual dependency parsing, taking dependency arcs as substructures: we project the predicted dependency arc distributions in the source language(s) to target language(s), and train a target language parser on the resulting distributions. Given an English tree bank as the only source of human supervision, SubDP achieves better unlabeled attachment score than all prior work on the Universal Dependencies v2.2 (Nivre et al., 2020) test set across eight diverse target languages, as well as the best labeled attachment score on six languages. In addition, SubDP improves zero shot cross-lingual dependency parsing with very few (e.g., 50) supervised bitext pairs, across a broader range of target languages.

## introduction zero-shot cross-lingual dependency parsing is the task that requires prediction of dependency parses without seeing any parsing example in the target language; instead, the model may use annotated parses in other languages. a popular line of work is annotation projection: the parses generated by a source language dependency parser are projected into the target language, where the projected parses are then used to train a new parser. as illustrated in figure 1b , most annotation projection methods typically output partial hard dependency trees, 1 where there either is or is not an arc between any pair of figure 1: illustration of subdp (top) vs. a representative of annotation projection (bottom; @xcite . an english parse tree, labeled with the universal dependencies conventions @xcite , is projected to the parallel chinese sentence. we denote dependency edges by arrows with the corresponding arc probabilities (if applicable) in parentheses, and word alignments by dashed lines. subdp can project either soft or hard trees, whereas most existing work only operates on hard trees. in this work, we introduce substructure distribution projection (subdp; figure 1a ), where dependency arcs act as substructures. we project substructure distributions, i.e., the conditional prob-ability distribution of the corresponding head given a word. 2 when the source parse is a hard tree, subdp has the same behavior as prior work (e.g., @xcite for arcs that are only involved in one-to-one alignments; for many-to-one alignments, subdp projects the corresponding arcs into soft arc distributions in the target language. therefore in subdp, a target language word may have multiple heads in the projected trees, where their probabilities sum to one. more generally, subdp may take dependency arc or label distributions (i.e., soft trees) in the source language(s), instead of hard trees, as the input. as in annotation projection approaches, the projected soft trees are then used to train a target language parser. we evaluate subdp on zero-shot cross-lingual dependency parsing with eight diverse languages from the universal dependencies v2.2 @xcite , where the english treebank is the only source of human supervision. taking english as the source language, subdp significantly outperforms all baseline methods on all distant languages (arabic, hindi, korean, and turkish) in our experiments, in terms of both labeled attachment scores (las) and unlabeled attachment scores (uas), while achieving superior uas on all nearby languages (german, french, spanish, and italian) as well. further analysis shows that subdp also helps improve zero-shot cross-lingual dependency parsing with a small amount of supervised bitext, across a broader range of target languages. these approaches make different assumptions on annotation availability, such as gold part-ofspeech tags @xcite @xcite , a reasonably good translator, which uses extra annotated data in the training process @xcite @xcite , high-quality bilingual lexicons @xcite @xcite , or language-specific constraints @xcite . most bitext-based work assumes annotated bitext @xcite @xcite 2021 ) only require minimal annotations (i.e., source language treebanks and unlimited raw text in relevant languages). we are mainly interested in the minimal annotation setting, and will compare to this line of work. our proposed method, subdp, falls into the category of annotation projection. some of the benefits of subdp relative to prior work are that it works well with minimal annotations, allows soft word alignment ( §3.2), supports both labeled and unlabeled parsing, and has a low time complexity o(n 2 ) for non-projective parsing. foot_2 subdp can be easily extended to other tasks, such as sequence labeling, where we can define substructures @xcite and substructure distributions. ## proposed approach: subdp our pipeline for zero-shot cross-lingual dependency parsing consists of three steps: (1) train a bi-affine dependency parser p 1 in the source language l 1 , (2) project annotations on l 1 sentences to their parallel sentences in the target language l 2 ( §3.3), and (3) train another bi-affine dependency parser p 2 for l 2 ( §3.4). we first present some background ( §3.1) and preliminaries ( §3.2). ## discussion our work is in line with recent work @xcite which shows that cross-lingual transfer can be done effectively with weak supervision such as wikipedia links. our results go further and study the setting of zero additional supervision beyond the source language treebank, demonstrating the potential of zero-shot cross-lingual dependency parsing with zero additional supervision, even between distant languages that do not share vocabulary or subwords. our work suggests a new protocol for dependency annotation of low-resource languages: (1) train a pretrained multilingual model following existing work such as xlm-r @xcite and criss @xcite , (2) annotate a small number of bitext pairs or generate bitext with trained unsupervised translation models, and (3) train a zero-shot cross-lingual dependency 25 35 45 55 65 75 85 avg. (all) 25 35 45 55 65 75 85 avg. (distant) 0 50 100 200 400 800 1.6k 3.2k 6.4k 12.8k 25.6k # bitext pairs 25 35 45 55 65 75 85 avg. (nearby) uas las figure 3: averaged las and uas on the universal dependencies v2.2 standard development set, with respect to the number of bitext pairs. for each language, we run 5 times with different random seeds. the x-axis is on a log scale. using zero bitext pairs corresponds to the direct transfer (dt; §4.2) baseline. all european languages are categorized as nearby languages, while the remaining are treated as distant languages. plots for individual languages can be found in appendix e. our contribution to zero-shot cross-lingual dependency parsing is arguably orthogonal to contextualized representation alignment @xcite , where pretrained multilingual language models are finetuned for better transfer. in contrast, we use the frozen pretrained models to extract features. in addition, projection quality controls by heuristic rule-based filtering @xcite may also be combined with subdp to further improve the performance. our results, on the other hand, demonstrate that multilingual pretrained models may have more applications beyond representation-based direct transfer-information extracted from these models without further supervision (e.g., word alignment in this work) may further benefit downstream tasks (e.g., zero-shot cross-lingual dependency parsing in this work) with appropriate usage. while this work depends on pretrained multilingual models such as criss @xcite , which require extensive computational resources to train from scratch, subdp may be applied whenever bitext alignment and cross-lingual word embeddings are available. in addition, the required pretrained cross-lingual models are useful for general purposes, and can be applied to other downstream nlp tasks. we suggest that subdp can be extended to other scenarios wherever relevant parallel signals are available, such as cross-lingual named entity recognition, cross-lingual constituency parsing, or zero-shot scene graph parsing for images using only the dependency supervision in text. we leave the further exploration of subdp on other tasks for future work.