On Evaluating Multilingual Compositional Generalization with Translated Datasets (2023)

Abstract: Compositional generalization allows efficient learning and human-like inductive biases. Since most research investigating compositional generalization in NLP is done on English, important questions remain underexplored. Do the necessary compositional generalization abilities differ across languages? Can models compositionally generalize cross-lingually? As a first step to answering these questions, recent work used neural machine translation to translate datasets for evaluating compositional generalization in semantic parsing. However, we show that this entails critical semantic distortion. To address this limitation, we craft a faithful rule-based translation of the MCWQ dataset from English to Chinese and Japanese. Even with the resulting robust benchmark, which we call MCWQ-R, we show that the distribution of compositions still suffers due to linguistic divergences, and that multilingual models still struggle with cross-lingual compositional generalization. Our dataset and methodology will serve as useful resources for the study of cross-lingual compositional generalization in other tasks.

## introduction a vital ability desired for language models is compositional generalization (cg), the ability to generalize to novel combinations of familiar units @xcite . semantic parsing enables executable representation of natural language utterances for knowledge base question answering (kbqa; @xcite . a growing amount of research has been investigating the cg ability of semantic parsers based on carefully constructed datasets, typically synthetic corpora (e.g., cfq; @xcite generated based on curated rules, mostly within monolingual english scenarios. as demonstrated by @xcite ?x0 wdt:p26 wd:q61597 . filter ( ?x0 != wd:q61597 )} figure 1 : example of neural machine translation (nmt, from mcwq, top) and rule-based translation (from mcwq-r, middle) from english to chinese. the compositions correctly captured by the translation system and the correspondences in the sparql query (bottom) are highlighted in the same color, while errors are in red. nmt often diverges semantically from the query: here, the compound "executive produce" is split. rbmt performs well due to awareness of grammar constituents. symbolic (e.g., rule-based) methodologies allow directly handling cg and were applied both to generate benchmarks @xcite @xcite and to inject inductive bias to state-of-the-art models @xcite . this motivates us to extend this idea to cross-lingual transfer of benchmarks and models. we propose to utilize rule-based machine translation (rbmt) to create parallel versions of mcwq and yield a robust multilingual benchmark measuring cg. we build an mt framework based on synchronous context-free grammars (scfg) and create new chinese and japanese translations of mcwq questions, which we call mcwq-r (multilingual compositional wikidata questions with rule-based translations). we conduct experiments on the datasets translated with gt and rbmt to investigate the effect of translation method and quality on cg in multilingual and cross-lingual scenarios. our specific contributions are as follows: ## multilingual compositional wikidata questions (mcwq) mcwq @xcite is the basis of our work. it comprises english questions inherited from cfq @xcite and the translated hebrew, chinese and kannada parallel questions based on google cloud translate, an nmt system. the questions are associated with sparql queries against wikidata, which were migrated from freebase queries in cfq. wikidata is an open knowledge base where each item is allocated a unique, persistent identifier (qid). foot_1 mcwq and cfq (and in turn, our proposed mcwq-r, see §4) share common english questions and associated sparql queries. mcwq introduces distinct multilingual branches, with the same data size across all the branches. due to the translation method employed in mcwq, it suffers from detrimental inconsistencies for cg evaluation (see figures 1 and 3 )-mainly due to the unstable mapping from source to target languages performed by nmt models at both the lexical and structural levels. we discuss the consequences with respect to translation quality in §4.3 and model performance in §6. ## mcwq-r: a novel translated dataset as stated in §2, data generation with gt disregards the "control over distribution", which is crucial for cg evaluation @xcite . thus, we propose to diverge from the mcwq methodology by translating the dataset following novel grammar of the involved language pairs to guarantee controllability during translation. such controllability ensures that the translations are deterministic and systematic. in this case, generalization is exclusively evaluated with respect to compositionality, avoiding other confounds. we create new instances of mcwq in japanese and chinese, two typologically distant languages from english, sharing one common language (chinese) with the existing mcwq. to make comprehensive experimental comparisons between languages, we also use gt to generate japanese translations (which we also regard as a part of mcwq in this paper), following the same method as mcwq. in this section, we describe the proposed mcwq-r dataset. in §4.1 we describe the pro-" questionpatternmodentities ": " did m1 's spouse executive produce m0 ", " questionwithbrackets ": " did [ erika mann ] 's spouse executive produce [ friedemann bach ]" , " questionpatternmodentities_zh ": " m1 的配偶主管是否生产了 m0 " " questionwithbrackets_zh ": "[ erika mann ] 的配偶执行官制作了 [ friedemann bach ] 吗", " sparqlpatternmodentities ": " ask where { m0 wdt : p1431 ? x0 . ? x0 wdt : p26 m1 . filter ( ? x0 != m1 )} " sparql ": " ask where { wd : q829979 wdt : p1431 ? x0 . ? x0 wdt : p26 wd : q61597 . filter ( ? x0 != wd : q61597 ) }" " questionpatternmodentities_zh ": " m1 的配偶执行制作了 m0 吗", " questionwithbrackets_zh ": "[ erika mann ] 的配偶执行制作了 [ friedemann bach ] 吗" figure 3 : example of an mcwq @xcite item in json format (top) and 2 fileds of the corresponding mcwq-r item (bottom). we present part of the fields: the english and sparql fields inherited from cfq and the chinese fields. specifically, we show an incorrectly translated example in mcwq where "excutive produce" is not translated as a composition while mcwq-r keeps good consistency with english. ## experiments while extensive experiments have been conducted on both the monolingual english @xcite and the gt-based multilingual benchmarks @xcite , the results fail to demonstrate pure multilingual cg due to noisy translations. consistent with prior work, we experiment in both monolingual and cross-lingual scenarios. specifically, we take into consideration both rbmt and gt branches foot_3 in the experiments for further comparison. ## conclusion in this paper, we introduced mcwq-r, a robustly generated multilingual cg benchmark with a proposed rule-based framework. through experiments with multilingual data generated with different translation methods, we revealed the substantial impact of linguistic differences and "translationese" on compositionality across languages. nevertheless, removing of all difficulties but compositionality, the new benchmark remains challenging both monolingually and cross-lingually. furthermore, we hope our proposed method can facilitate future investigation on multilingual cg benchmark in a controllable manner.