Improving Zero-Shot Cross-lingual Transfer for Multilingual Question Answering over Knowledge Graph (2021)

Abstract: Multilingual question answering over knowledge graph (KGQA) aims to derive answers from a knowledge graph (KG) for questions in multiple languages. To be widely applicable, we focus on its zero-shot transfer setting. That is, we can only access training data in a high-resource language, while need to answer multilingual questions without any labeled data in target languages. A straightforward approach is resorting to pre-trained multilingual models (e.g., mBERT) for cross-lingual transfer, but there is a still significant gap of KGQA performance between source and target languages. In this paper, we exploit unsupervised bilingual lexicon induction (BLI) to map training questions in source language into those in target language as augmented training data, which circumvents language inconsistency between training and inference. Furthermore, we propose an adversarial learning strategy to alleviate syntax-disorder of the augmented data, making the model incline to both language- and syntax-independence. Consequently, our model narrows the gap in zero-shot cross-lingual transfer. Experiments on two multilingual KGQA datasets with 11 zero-resource languages verify its effectiveness.

## introduction with the advance of large-scale human-curated knowledge graphs (kg), e.g., dbpedia @xcite and freebase @xcite , question answering over knowledge graph (kgqa) has become a crucial natural language processing (nlp) task to answer factoid questions. it has been integrated into real-world applications like search engines and personal assistants, so it attracts more attention from both academia and industry @xcite @xcite . recently, a rising demand of kgqa systems is to answer the multilingual questions, motivating us to focus on multilingual kgqa. however, building a large-scale kg, as well as annotating qa data, is costly for each new language, not to mention many minority languages with a few native annotators. therefore, we adopt a zero-shot cross-lingual transfer setting -a kgqa model is developed to perform inference on multilingual questions with the only access to training data and associated kg in a high-resource language (e.g., english). providing the success of pre-trained monolingual encoders @xcite , some works (e.g., mbert @xcite and xlm-r @xcite ) pre-train a transformer encoder @xcite on largescale non-parallel multilingual corpora in a selfsupervised manner. then given an nlp task, a general paradigm for zero-shot cross-lingual transfer is to fine-tune a pre-trained multilingual encoder on the data in a data-rich (source) language. and the fine-tuned model is generalizable enough to perform inference in other low-resource (target) languages with surprising quality of prediction. this paradigm can be adapted to kgqa to build symbolic logical forms (e.g., query graph @xcite ) for kg query. however, it is witnessed that there is a considerable kgqa performance gap between source and target languages, which is consistent with the empirical results on a wide range of other tasks by prior works @xcite . to bridge the gap, translation approaches are proven effective on multilingual benchmarks @xcite . as a way of data augmentation, they perform source-to-target translation to obtain multilingual training data. further with advanced techniques @xcite , they achieve state-of-the-art effectiveness. but these approaches rely heavily on a well-performing translator. the translator is not always available especially for a minority language since its training requires a large volume of parallel bilingual corpus. therefore, to be applicable to more languages, we assume that neither translators nor parallel corpora are available in this work. in this paper, to adapt the translation approaches in our zero-resource scenario, we naturally propose to replace the full-supervised machine translator with unsupervised bilingual lexicon induction (bli) for word-level translation. specifically, as in prior works @xcite , a bli model is first trained on non-parallel bilingual corpora. then, via bilingual word alignments in bli, we map the training questions in source language into those in target languages to obtain augmented multilingual training data. consequently, even simply learning a kgqa model on the augmented data can circumvent language inconsistency between training and inference and thus bridge the performance gap in zero-shot crosslingual transfer. to explain why bli is competent, it is observed that kgqa mainly involves phraselevel semantics @xcite . compared to other tasks depending on sentence-level contextualization, kgqa is insensitive to long-term dependency but benefits from the language consistency. moreover, we propose an adversarial strategy to mitigate the syntax-disorder caused by bli. specifically, we present a discriminator on top of the encoder, which is trained to distinguish whether the input is a grammatical question in source language or a bli-translated one in target language. meanwhile, jointly with kgqa goal, the encoder is fine-tuned to fool the discriminator so that the questions' representations are both language-and syntax-agnostic. so the trained kgqa model is robust to syntax-disorder and becomes insensitive to the question language, leading to superior performance on multilingual kgqa. experiments conducted on two multilingual kgqa datasets with 11 zero-resource languages verify the effectiveness of our approach. ## kgqa task definition we give a background of monolingual kgqa, followed by multilingual kgqa and its data format. ## methodology this section starts with a base framework for monolingual kgqa, followed by our proposed multilingual solutions. lastly, details about training and inference are elaborated. ## related work there are mainly two categories of approaches to handle monolingual question answering over knowledge graph (kgqa) task. (1) information retrieval-based approaches align a question with its answer candidates in the same semantic space, where the candidates usually stem from kg neighbors of the topic entity detected in the questions @xcite @xcite @xcite . (2) semantic parsing-based approaches first translate a question into the corresponding logical form, e.g., program @xcite or query graph @xcite @xcite @xcite @xcite , and then execute the logical form over kg to derive the final answer. note a logical form is usually composed of a series of grammars or operators pre-defined by experts. this paper is in line with the second category to generate query graph for kg execution. to the best of our knowledge, there are only few works targeting multilingual kgqa @xcite , which rely on extensive multilingual training data with hand-crafted features while are inapplicable to the zero-shot transfer scenario. so we adopt the pipeline by @xcite for monolingual scenario as our base model but update the encoders with the transformer @xcite to strengthen their expressive power and facilitate recent pre-trained multilingual initializations. given task-specific data in a source language, cross-lingual models are trained to perform inference in target languages in a low-or zero-resource scenario. typically, cross-lingual models are proposed in two paradigms. 1) universal encodingbased paradigm represents multilingual natural language text into language-agnostic embeddings the same semantic space. early works focus on aligning multilingual word embedding @xcite @xcite , while recent efforts are mainly made on large-scale pre-trained multilingual encoder, such as mbert @xcite , xlm @xcite , unicoder @xcite , xlm-r @xcite , and alm @xcite . they can perform zero-shot cross-lingual transfer by training in the source language while directly inference in target language. 2) translation-based paradigm employs well-trained machine translators to map the training or test examples in source language to those in target translation. recent common practice tends to leverage the second paradigm to generate multilingual data to narrows the zero-shot cross-lingual performance gap in the first paradigm, which leads to state-of-the-art results on several cross-lingual benchmarks. in contrast, we consider a zero-resource scenario where translators are unavailable and we thus resort to unsupervised bli in light of kgqa's characteristics. as a branch of universal encoding at word level, bilingual lexicon induction (bli) (a.k.a crosslingual word embedding -clwe) is learned to align bilingual word embeddings in the same space, where the embeddings are pre-trained on monolingual corpora and the alignment is trained in either a (semi-)supervised or unsupervised manner @xcite @xcite @xcite @xcite @xcite . to alleviate "hubness" problem @xcite in bli, alternatives of the distance measurement are proposed to substitute nearest neighbor (nn) during the alignment, such as inverted-softmax @xcite and csls @xcite . in addition to building bilingual dictionary via word-level translation, a well-trained bli model can serve as a weak baseline of sentence-level translation @xcite , a seed model for unsupervised translation @xcite or a bilingual variant of copy mechanism in summarization @xcite . moreover, adversarial training is usually integrated into cross-lingual models for languageagnostic representation learning, such as unsupervised bli @xcite , unsupervised translation @xcite , cross-lingual sequence labeling @xcite and cross-lingual classification @xcite . in contrast, our adversarial strategy not only considers languageagnostic representations but also aims at making the model insensitive to syntax-disorder and thus competent in zero-resource scenario. ## conclusion we propose a novel approach for zero-shot crosslingual transfer in multilingual kgqa, which augments training data by bilingual lexicon induction, and leverages a syntax-agnostic adversarial learning strategy to alleviate the syntax-disorder problem caused by bli. experimental results on two multilingual kgqa datasets in 11 zero-resource languages verify its effectiveness.