Learning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension (2022)

Abstract: Multilingual pre-trained models are able to zero-shot transfer knowledge from rich-resource to low-resource languages in machine reading comprehension (MRC). However, inherent linguistic discrepancies in different languages could make answer spans predicted by zero-shot transfer violate syntactic constraints of the target language. In this paper, we propose a novel multilingual MRC framework equipped with a Siamese Semantic Disentanglement Model (S2DM) to disassociate semantics from syntax in representations learned by multilingual pre-trained models. To explicitly transfer only semantic knowledge to the target language, we propose two groups of losses tailored for semantic and syntactic encoding and disentanglement. Experimental results on three multilingual MRC datasets (i.e., XQuAD, MLQA, and TyDi QA) demonstrate the effectiveness of our proposed approach over models based on mBERT and XLM-100.

## introduction multilingual pre-trained language models (plms) @xcite @xcite have been widely explored in cross-lingual understanding tasks. however, zero-shot transfer method based on multilingual plms does not work well for low-resource language mrc. such multilingual mrc models could roughly detect answer spans but may fail to predict the precise boundaries of answers @xcite . in order to address this issue, existing methods mainly resort to external resources. based on the finding that 70% of answer spans are languagespecific phrases (e.g., named entities, noun phrases) in mlqa @xcite , @xcite propose an additional language-specific knowledge [question] when were attacks on teachers the highest? @xcite between 2005 and 2010 [the answer is a prepositional phrase containing a noun phrase] root vp np pp dt jj in np cd an high between 2005 [passage] [bipar] 韦小宝见蛾眉刺上闪出的月光不住晃动，有时直掠到他脸上[...] as trinket watched, the gleam of reflected light from the stiletto began to jump and waver.[...] [question] 什么在晃动？ what's shaking? [answer -ground truth] 月光 the gleam of reflected light [answer -model prediction] 月光不住 [the answer violating syntactic constituent boundaries] @xcite where the ground-truth answer is a syntactic constituent. (b) a case from bipar @xcite where the answer predicted by a model transferred from english to chinese violates syntactic constituent boundaries in the target language. on the bilingual parallel mrc corpus bipar @xcite , we have compared two mrc models: a monolingual mrc model trained on the chinese data of bipar vs. an mbert-based mrc model trained on the english data of bipar and adapted to chinese via zero-shot transfer. for questions where the monolingual model correctly predicts the answer and respect syntactic constraint, 23.15% of them are incorrectly predicted by the transfer model, and the predicted answers violate the syntactic constraint, illustrated by the case in figure 1 @xcite . this suggests that the source language syntax may have a negative impact on the answer boundary detection in the target language during zero-shot transfer, due to the linguistic discrepancies between the two languages. however, linguistic discrepancies are diverse and it is difficult to learn them. we hence propose to decouple semantics from syntax in pretrained models for multilingual mrc, transforming the learning of linguistic discrepancies into universal semantic information. specifically, we propose a siamese semantic disentanglement model (s 2 dm) that utilises two latent variables to learn semantic and syntactic vectors in multilingual pretrained representations. as shown in figure 2(a) , stacking a linear output layer for mrc over the disentangled semantic representation layer, we can fine-tune the multilingual plms on the richresource source language and transfer only disentangled semantic knowledge into the target language mrc. our model aims to reduce the negative impact of the source language syntax on answer boundary detection in the target language. to disassociate semantic and syntactic information in plms well, we introduce objective functions of learning cross-lingual reconstruction and semantic discrimination together with losses of incorporating word order information and syntax structure information (part-of-speech tags and syntax parse trees). we use a publicly available multilingual sentence-level parallel corpus with syntactic labels to train s 2 dm. to summarize, our main contributions are as follows. ## approach figure 2 shows the architecture of our multilingual mrc framework with the proposed siamese semantic disentanglement model. ## conclusions in this paper, we have presented a novel multilingual mrc model for zero-shot cross-lingual transfer, which can disentangle semantic from syntactic representations and explicitly transfer semantic information from rich-resource language to low-resource languages, reducing the influence of syntactic differences between languages on the answer span prediction of the target language. to disassociate semantics from syntax in multilingual pre-trained representations, we propose the siamese semantic disentanglement model that semantics/syntax-oriented losses to guide latent variables to learn corresponding information. for low-resource languages without training data for the decoupling model, our theoretical analysis and experiments verify the generalization of our multilingual mrc model. further in-depth analyses suggest that the proposed s 2 dm can efficiently disentangle semantics from syntax and significantly improve syntactic consistency of answer predictions on the target language after zero-shot crosslingual transfer.