Fine-Grained Analysis of Cross-Linguistic Syntactic Divergences (2020)

Abstract: The patterns in which the syntax of different languages converges and diverges are often used to inform work on cross-lingual transfer. Nevertheless, little empirical work has been done on quantifying the prevalence of different syntactic divergences across language pairs. We propose a framework for extracting divergence patterns for any language pair from a parallel corpus, building on Universal Dependencies. We show that our framework provides a detailed picture of cross-language divergences, generalizes previous approaches, and lends itself to full automation. We further present a novel dataset, a manually word-aligned subset of the Parallel UD corpus in five languages, and use it to perform a detailed corpus study. We demonstrate the usefulness of the resulting analysis by showing that it can help account for performance patterns of a cross-lingual parser.

## introduction the assumption that the syntactic structure of a sentence is predictably related to the syntactic structure of its translation has deep roots in nlp, notably in cross-lingual transfer methods, such as annotation projection and multi-lingual parsing @xcite @xcite , as well as in syntaxaware machine translation (mt; @xcite @xcite . relatedly, typological parameters that provide information on the dimensions of similarity between grammars of different languages were found useful for a variety of nlp applications @xcite . for example, neural mt in low-resource settings has been shown to benefit from bridg- tomseuni gigohan nonmuneun the article by thompson thompson-contribute-article-nom attr topic nsubj acl:relcl root root det nmod case figure 1: an example en-ko sentence pair exhibiting a divergence, where an en nmod path corresponds to a ko acl:relcl+nsubj path (paths are given in bold red). the en preposition by is not considered a content word and is not aligned with the ko verb gigohan. see § 3 for further details. nevertheless, little empirical work has been done on systematically quantifying the type and prevalence of syntactic divergences across languages. moreover, previous work generally classified divergences into a small set of divergence classes, often based on theoretical considerations @xcite or on categorical ("hard") typological features selected in an ad-hoc manner, and left basic questions, such as how often pos tags are preserved in translation and what syntactic structures are likely correspondents of different syntactic relations, largely unaddressed. see § 2. we propose a language-neutral, fine-grained definition of cross-linguistic morphosyntactic divergences (clmd) that allows for their extraction using a syntactically annotated, content-wordaligned parallel corpus. concretely, we classify clmd based on the edge labels on the dependency paths between corresponding pairs of content words ( § 3.2). see figure 1 for an example. 1 1 it may appear that divergences recoverable by means of ud edge labels are purely syntactic and not morphosyntactic. however, this is not the case: the domain of "pure syntax" is not well defined in a non-theoretical perspective, and many phenomena we are dealing with, e.g. a switch from a direct-we further conduct a detailed corpus study, manually aligning content words in a subset of the pud corpus @xcite over five language pairs-english-french (en-fr), english-russian (en-ru), english-chinese (en-zh), english-korean (en-ko), and english-japanese (en-jp) ( § 3.1)-and analyze the prevalence of divergences by types ( § 4). the resulting resource can be useful for mt research, by guiding the creation of challenge sets focusing on particular constructions known to be cross-linguistically divergent, as well as by guiding preprocessing of source-side sentences for better mt performance. 2 the emerging clmd provide information not only on the macro-structure of the grammar (e.g., whether the language is pro-dropping), but also on parameters specific to certain lexical classes (e.g., modal verbs) and probabilistic tendencies (e.g., japanese tends to translate sequences of events, expressed in english using coordinating conjunctions, with subordinate clauses). see § 5. further experiments demonstrate the methodology's applicative potential. first, we show that the proposed methodology can be straightforwardly automated by replacing manual parses and alignments with automatically induced ones ( § 7). we present a study done on a larger en-zh corpus, which yields results similar to those obtained manually. secondly, we show that the reported distribution over divergence types is predictive of the performance patterns of a zero-shot parser ( § 8). ## related work comparing syntactic and semantic structures over parallel corpora is the subject of much previous work. @xcite compiled a multiplyparallel corpus and annotated it with increasingly refined categories in an attempt to abstract away from syntactic detail but did not report any systematic measurement of the distribution of divergences. @xcite , @xcite , @xcite , and @xcite studied divergences over semantic graphs and argument-structure phenomena, while a related line of work examined divergences in discourse phenomena @xcite . other works studied the ability of a given grammar formalism to capture clmd in a parallel corpus (e.g., object construction to an oblique construction, often involve morphological processes, such as adding a case ending. @xcite . however, none of these works defined a general methodology for extracting and classifying clmd. the only previous work we are aware of to use ud for identifying clmd is @xcite , which addresses mandarin-cantonese divergences by comparing the marginal distribution of syntactic categories on both sides (without alignment). relatedly, deng and xue (dx17; 2017) aligned phrase structure trees over an en-zh parallel corpus. notwithstanding the similarity in the general approach, we differ from dx17 in (i) specifically targeting content words, (ii) relying on ud, which is standardized cross-linguistically and allows to simplify the alignment process by focusing on the level of words, foot_1 and (iii) addressing multiple language pairs. it should be noted that the classification of divergences presented in dx17 is rather coarse-grained. of the seven classes in their study, five (transitivity, absence of function words, category mismatch, reordering, and dropped elements) reflect local syntactic differences; one (lexical encoding) covers many-to-one/one-to-many alignments and non-literal word translations; and the remaining residual type (structural paraphrase) indiscriminately covers more substantial clmd. we address this limitation and propose a methodology that automatically derives fine-grained clmd from aligned annotated corpora and enables straightforward computation of their type statistics. ## fine-grained classification of clmd in this section, we present a novel cross-linguistic dataset that provides a high-resolution overview of morphosyntactic differences between pairs of languages and a formal definition of morphosyntactic divergences formulated based on it. divergences in the syntax of sentences and their translations can stem from a number of reasons. setting aside semantic divergences, which are differences in the content expressed by the source and the target @xcite , the remaining varieties of divergences are essentially different ways to express the same content @xcite , which we call clmd. we define clmd empirically to be recurrent divergence patterns in the syntactic structures of sentences and their translations. while content differences may account for some of the observed syntactic divergences, by aiming for recurring patterns we expect to filter out most such cases, as they are subject to fewer grammatical constraints and should thus not yield systematic patterns of morphosyntactic divergence. it is harder to distinguish between translation artifacts and clmd in translated sentences that are due to the genuine differences between grammar and usage. however, translated texts are usually characterized by a higher degree of morphosyntactic transfer and rarely portray the target language as more different from the source language than it needs to be @xcite . therefore, we do not expect to find spurious recurrent morphosyntacticdivergence patterns introduced by the process of translation. ## empirical study of divergences we apply the proposed methodology to the aligned pud. we compare syntactic relations, analyzing correspondences between pos tags as well as cor-respondences between single-edge relations in english and target-side dependency paths. ## qualitative analysis of divergences in this section, we analyze prominent cases of divergences revealed by applying our method, attempting to demonstrate how fine-grained clmd may be detected from the confusion matrices and shedding light on what challenges are involved in bridging these divergences (e.g., for the purposes of mt or cross-lingual transfer). some of the divergences arise due to real differences between grammars; others are largely due to inconsistent application of the ud methodology. ## revisiting dorr's divergences we turn to show that the types of the seminal classification of divergences from @xcite can be, with a single exception, recast in terms of ud divergences. we quote the original formulations of the divergences illustrated through english-spanish or english-german examples. thematic divergence: e: i like mary ⇔ s: maría me gusta a mí 'mary pleases me.' in ud, this corresponds to the situation when the original obj or obl becomes the nsubj and vice versa. the divergence will correspond to a csr of type (nsubj, obj) or (nsubj, obl). a "full" thematic divergence will also involve the inverse divergence (obj, subj) or (obl, subj). 8 the list of examples we can discuss goes on. for example, while investigating the cross-linguistic patterning of english advcl, we noticed that it often gets mapped to ccomp in french and acl in russian. both divergent annotations seem to be erroneous as the sentences they appear in are covered by the definition of advcl provided in the ud manual. however, the french case is interesting in that the source advcl in question can be characterized semantically: instead of denoting a secondary action, they reflect a sequence of events or parallel scenes (e.g., columbus sailed across the atlantic... sparking a period of european exploration of the americas). another problem is presented by multi-word expressions analyzed as proper nouns where all tokens have the same pos tag. the ud manual advises to retain the original parts of speech in proper nouns consisting of phrases (e.g., cat on a hot tin roof ) but allows to treat "words that are etymologically adjectives" as propn in names such as the yellow pages. when such names are translated, propn get reanalyzed as adj, noun, etc., producing spurious clmd. promotional divergence: e: john usually goes home ⇔ s: juan suele ir a casa 'john tends to go home.' this corresponds to the situation where the original root predicate becomes an xcomp, and the original advmod takes its place as the root. corresponding csr type: (advmod, xcomp). demotional divergence: e: i like eating ⇔ g: ich esse gern 'i eat likingly.' the original xcomp becomes the root predicate, and the original root predicate is demoted to the position of an advmod. the relevant csr type is (xcomp, advmod). structural divergence: e: john entered the house ⇔ s: juan entró en la casa 'john entered in the house.' the original obj becomes an obl. csr type: (obj, obl). conflational divergence: e: i stabbed john ⇔ s: yo le di puñaladas a juan 'i gave knife-wounds to john.' the original root predicate is in a oneto-many alignment with a combination of a root predicate and its obj. categorial divergence: e: i am hungry ⇔ g: ich habe hunger 'i have hunger.' the original root predicate becomes an obj. csr type: (nsubj, nsubj + obj). lexical divergence: e: john broke into the room ⇔ s: juan forzó la entrada al cuarto 'john forced (the) entry to the room.' divergences of this type arise whenever aligned words have at best partially overlapping semantic content and never appear on their own but always with other divergences. the information necessary to ascertain the degree of word-meaning overlap is not embedded into ud or any other cross-lingual annotation scheme; therefore we were unable to provide a formal interpretation of this type of divergence. frequencies of dorr's divergences in pud are presented in ## perspectives for automation one of the strengths of our approach is that it only relies on ud parses and alignments, for which automatic tools exist in many languages. to demonstrate the feasibility of an automated protocol, we conducted an analysis of the wmt14 en-zh news commentary corpus. 9 we used tsinghuaaligner @xcite and pretrained english and chinese ud parsers from the stanfordnlp toolkit @xcite . to verify that the aligner we are using is adequate for the task, we aligned the en-zh pud corpus pair and checked the resulting precision and recall of the edges corresponding to content words. 10 the results (@xmath0.86, @xmath1.32) indicate that the automated approach is able to recover around a third of the information obtained through manual alignment with reasonable precision. importantly, we find recall to be nearly uniform for all source edge types, which suggests that the low recall can be mitigated by using a larger corpus without biasing the results. the pos and edge-type confusion matrices built from this experiment are very similar to the ones reported in this paper (save for compound, which is not produced by the stanford zh parser), and are not reproduced here (they can be found in the supplementary materials). ## applicability for zero-shot parsing we come to demonstrate the applicability of our method for analyzing the performance of a downstream cross-lingual transfer task. we consider zero-shot cross-lingual parsing @xcite as a test case and investigate to what extent the performance of a zeroshot parser on a given dependency label can be predicted from its stability in translation. as test sets we use the test sets of gsd ud corpora for the five languages (ru, fr, zh, ko, and jp), as well as the corresponding pud corpora. we train a parser following the setup of @xcite and use a pretrained multilingual bert @xcite , feeding its output embeddings into a biaffine-attention neural ud parser @xcite trained on the english ewt corpus. we evaluate the parser's ability to predict relation types by computing f-scores for each de-pendency label (save for labels corresponding to function words that were generally not aligned). appendix e gives full implementation details. we start by computing spearman correlations between f-scores and the preservation indices, defined as the proportion of identity mappings in the confusion matrices for each corpus (e.g., preservation for acl in ru is 0.48, while in jp it is 0.37). the correlations are very strong for some languages and noticeable for others (ρ = 0.62, 0.75, 0.31, 0.42, 0.77 for ru, fr, zh, ko, and jp respectively on gsd test sets, and ρ = 0.7, 0.82, 0.72, 0.84, 0.68 on pud). we hypothesize that the preservation of a relation in translation is related to the ability of a zero-shot parser to predict it. in order to control for obvious covariates, we introduce two control variables: (1) source-side hardness (test-set f-scores attained by the parser on english dependency relations) and (2) target-side hardness (f-scores attained by a parser trained on the targetlanguage ud gsd corpus on the target-language test set). we use a mixed-effects model with preservation, source-side hardness, and target-side hardness as fixed effects, random intercepts for language, and f-scores for dependency relations as the dependent variable. we then used likelihood-ratio test to compute p-values for the difference in predictive power between the model without preservation and one with it. the p-value (using holm correction) is highly significant (< 0.001) for the pud corpora, and for gsd it is significant with @xmath2.02. these results suggest that morphosyntactic differences between languages, as uncovered by our method, play a role in the transferability of parsers across languages. this also underscores the potential utility of bridging clmd for improving syntactic transfer across languages. ## discussion the presented methodology gives easy access to different levels of analysis. on one hand, by focusing on content words, the approach abstracts away from much local-syntactic detail (such as reordering or adding/removing function words). at the same time, the methodology and datasets provide means to investigate essentially any kind of well-defined clmd. indeed, since function words in ud tend to be dependents of content words, we may analyze the former by considering the distri-bution of function word types that each type of content word has. moreover, sub-typing dependency paths based on their linear direction can allow investigating word-order differences. 11 other than informing the development of crosslingual transfer learning, our analysis directly supports the validation of ud annotation. for example, we reveal inconsistencies in the treatment of multi-word expressions across languages. thus, the translation of many nps with adjectival modifiers, such as celtic sea or episcopal church, are analyzed as compounds. languages such as ru, lacking a truly productive nominal-compound relation, carve this class up based mostly on the pos of the dependent element (e.g., episcopal corresponds to a ru amod), its semantic class (e.g., compounds with cardinal directions are ru amods), and whether the dependent element itself has dependents (these mostly correspond to ru nmods). our method can be used to detect and bridge such inconsistencies. in conclusion we note that our analysis suggests that considerable entropy in the mapping between the syntactic relations of the source and target sides can be reduced by removing inconsistencies in the application of ud, and perhaps more importantly by refining ud with semantic distinctions that will normalize corresponding constructions across languages to have a similar annotation. this will simultaneously advance ud's stated goal of "bringing out cross-linguistic parallelism across languages" and, as our results on zero-shot parsing suggest, make it more useful for cross-linguistic transfer.