On the Relation between Syntactic Divergence and Zero-Shot Performance (2021)

Abstract: We explore the link between the extent to which syntactic relations are preserved in translation and the ease of correctly constructing a parse tree in a zero-shot setting. While previous work suggests such a relation, it tends to focus on the macro level and not on the level of individual edges—a gap we aim to address. As a test case, we take the transfer of Universal Dependencies (UD) parsing from English to a diverse set of languages and conduct two sets of experiments. In one, we analyze zero-shot performance based on the extent to which English source edges are preserved in translation. In another, we apply three linguistically motivated transformations to UD, creating more cross-lingually stable versions of it, and assess their zero-shot parsability. In order to compare parsing performance across different schemes, we perform extrinsic evaluation on the downstream task of cross-lingual relation extraction (RE) using a subset of a standard English RE benchmark translated to Russian and Korean. In both sets of experiments, our results suggest a strong relation between cross-lingual stability and zero-shot parsing performance.

## introduction recent progress in cross-lingual transfer methods, such as multi-lingual embeddings @xcite , enabled significant advances in a wide range of cross-lingual natural language processing tasks. the transferred models, however, are not uniformly effective in addressing languages with different grammatical structures, and little is known about the settings under which cross-lingual transfer is more or less effective. a prominent way of facilitating transfer of grammatical knowledge from one language to another figure 1 : a ud parse of the english phrase japanese company (left) and of its translations to arabic (middle) and japanese (right). word order aside, the ud tree of the arabic translation is identical to the english one while that of the japanese translation is different. the amod edge in this phrase is stable in translation to arabic but unstable in translation to japanese. while these schemes abstract away from many syntactic differences, there is still considerable variability in the strategies employed to express the same basic meanings across languages. in this work, we are mainly interested in the flip side of variability, viz. the stability of a given scheme-the extent to which its annotations are invariant under translation. see an example in fig. 1 . we present two sets of empirical findings that establish a strong relation between stability and success of zero-shot (zs) cross-lingual transfer from a single language in a multiply parallel corpus setting where no annotated data from the target languages were used for training. such a setting is a natural starting point for our investigation, as it is both practically useful (see, e.g., @xcite @xcite for successful examples of employing zs learning cross-lingually) and is based on a homogeneous training set, which minimizes the risk of introducing confounds into the analysis. the first set of experiments quantifies the effect of stability of edges in ud parse trees on a zs parser's performance. in order to check if a particular edge was stably transferred from the corresponding source sentence, we use an extended version of the manually aligned subset of the parallel ud dataset @xcite , which provides word-aligned translations of circa 1000 english sentences into six languages with different typological profiles. we find highly consistent trends across all language pairs, where stable edges receive an average labeled attachment score (las; a standard evaluation metric in dependency parsing) that is often twice or more bigger than the one for edges that do not correspond to a source side edge. these findings indicate a strong link between the stability of an edge and its contribution to zs parsing quality and suggest that zs parsing performance can be enhanced by improving the stability of the underlying syntactic representation. our next set of experiments are the first steps in this direction. concretely, we define three transformations, each targeting an area of syntax that is known to give rise to cross-lingual divergences in ud terms, and thus create three slightly modified versions of ud. we then apply these transformations to the training set in order to check if these versions of ud lead to improved zs performance. as attachment scores across different schemes are not comparable, we opt for extrinsic evaluation through zs cross-lingual relation extraction. since there are no multilingual re datasets that include data from non-western-european languages, we translated a 500-sentence subset of the english tacred dataset @xcite into korean and russian and annotated it with the relations from respective english source sentences. our results show that modified versions of ud give rise to improved results. this indicates that ud can be made more cross-lingually stable without sacrificing its usefulness for downstream tasks. the paper is organized as follows. in §2, we explore the correlation between zs performance and stability. in §3, we present transformations to ud annotations ( §3.1) and the methodology for comparing downstream usefulness of vanilla ud and transformed ud ( §3.2). our experimental setup is described in §4, and the results are presented in §5. related work is summarised in §6. section 7 concludes the paper. ## edge stability and zs parsability this section evaluates the relation between the extent to which an edge in a translated sentence corresponds to an edge in the original sentence (which we operationalize as several stability categories) and the ability of a zs parser to parse it correctly (its zs parsability). we use the manually aligned subset of the parallel ud corpus (n20; @xcite , which augments the pud dataset @xcite with alignments between corresponding content words over five language pairs, the source language being english (en), and the target languages being french (fr), russian (ru), japanese (jp), chinese (zh), and korean (ko). we further use an extension of the corpus to an additional language pair, english-arabic (en-ar), in order to increase the diversity of examined languages @xcite . 2 in all cases, both the ud annotation and the alignment were done manually. we next train a state-of-the-art zs parser on en (same as used for the experiment described in §3.2) and examine its performance on pud over the six target languages. we partition the edges in the target-language test parses into categories based on their stability and investigate the performance of the parser on an edge as a function of its stabilitycategory membership. ## comparing zero-shot performance across representation schemes in this section, we aim to manipulate the annotation scheme so as to increase its stability and test whether the modification yields more crosslingually useful annotations. we achieve this by devising three linguistically motivated, language agnostic transformations ( §3.1) and applying them to the downstream task of zs relation extraction ( §3.2). while some previous work proposed syntactic preprocessing of the source-side ud trees for the sake of cross-lingual zs parsing, we are not aware of any previous work that compares the performance on a parsing-dependent downstream task across different schemes. ## experimental setup ud parser. we use allennlp's @xcite implementation of the deep biaffine attention graph-based model of @xcite . we replace the trainable glove embeddings with the pre-trained multilingual bert @xcite ) embeddings 7 provided by hugging face @xcite . we also replace the bil-stm encoder with self-attention to increase the parser's cross-lingual transfer capabilities @xcite . finally, we do not use gold (or any) pos tags to represent a more realistic scenario and avoid introducing the potentially confounding factor of pos divergence into our analysis. we use al-lennlp's default settings and hyper-parameters (all hyper-parameter values are given in appendix c). ## related work the relation between stability and cross-lingual transfer has been the subject of previous work. however, the great majority of it has focused on word order @xcite @xcite and morpho- +3 +3.7 +3.2 +7.9 +8.9 +1.5 +4.6 +5.6 predicate 71.5 24.7 36.6 88.8 25.1 39.1 72.4 29.6 41.9 89.8 31.1 46.1 0 +3.1 +3.6 -1.3 +1.1 +1.2 +0.9 +6.9 +7.6 +0.8 +6.9 +8 oblique 75.9 25.2 37.6 90.7 24.4 38.4 74.2 29.3 41.8 89.9 30.7 45.8 +4.4 +3.6 +4.6 +0.6 +0.4 +0.5 +2.7 +6.6 +7.5 +0.9 +6.5 +7.7 2018 ) also demonstrated the benefits of modifying ud parse trees in order to improve their utility for cross-lingual applications. these modifications, however, took ud for granted and only targeted specific types of subtrees to make them look more like the corresponding subtrees in the target language (e.g., by converting an english possessive construction i have x into a more arabic-looking locative-possessive construction is x at me). this approach, therefore, is highly language-pair specific as the transformations de-fined on the differences between surface syntax of english and arabic will not be useful when presented with another target language. our work, on the other hand, unconditionally altered the scheme itself, and models using it can be profitably transferred to any target language. others @xcite @xcite @xcite also proposed transformations aimed at emphasizing useful connections between tokens but not in a cross-lingual context. ## conclusion our work establishes a strong relationship between stability and cross-lingual transfer, even at the level of individual edges. we find that the stability of an edge is a strong indicator for the ability of a zs parser to predict it, suggesting that despite recent advances in zs parsing and cross-lingual embeddings, these models still face difficulties in generalizing over the grammars and syntactic-usage patterns. furthermore, we show that it is possible to improve the annotation stability of representations using linguistically motivated and universally applicable transformations, which lead to better cross-lingual transferability. our results suggest several directions for future work in terms of designing more transferable annotation schemes and improving evaluation practices for zs parsing. they also suggests a path towards a theory of the relation between the linguistic properties of a construction and the ability to effectively process it using cross-lingual-transfer tools. tao meng, nanyun peng, and kai-wei chang. 2019. target language-aware constrained inference for cross-lingual dependency parsing. the results are presented in