Fine-grained Multi-lingual Disentangled Autoencoder for Language-agnostic Representation Learning (2022)

Abstract: Encoding both language-specific and language-agnostic information into a single high-dimensional space is a common practice of pre-trained Multi-lingual Language Models (pMLM). Such encoding has been shown to perform effectively on natural language tasks requiring semantics of the whole sentence (e.g., translation). However, its effectiveness appears to be limited on tasks requiring partial information of the utterance (e.g., multi-lingual entity retrieval, template retrieval, and semantic alignment). In this work, a novel Fine-grained Multilingual Disentangled Autoencoder (FMDA) is proposed to disentangle fine-grained semantic information from language-specific information in a multi-lingual setting. FMDA is capable of successfully extracting the disentangled template semantic and residual semantic representations. Experiments conducted on the MASSIVE dataset demonstrate that the disentangled encoding can boost each other during the training, thus consistently outperforming the original pMLM and the strong language disentanglement baseline on monolingual template retrieval and cross-lingual semantic retrieval tasks across multiple languages.

## introduction pre-trained multilingual language models such as multilingual bert (mbert) @xcite and xlm-roberta (xlm-r) @xcite have been extensively explored and used in academia and industry. these models encode both language-specific information (e.g., grammar, tense, syntax) and language-agnostic information (e.g., semantic, entity, sentiment) into one highdimensional embedding. however, it has been demonstrated that such encoders perform poorly in some tasks due to a lack of capacity to disentangle fine-grained language-agnostic and language- * *this work is finished during the internship at amazon alexa ai specific information. @xcite @xcite @xcite . in this work, we proposed a lightweight encoding architecture called fine-grained multilingual disentangled autoencoder (fmda) that can disentangle semantic representations at different aspects. the training of proposed encoder adopts reconstruction loss and contrastive learning. the contributions of our proposal are as follows: application tasks utterances language semantic template slot cross-lingual retrieval source: wecke mich um [time: fünf uhr] auf different same same/similar same target: wake me up at [time: five am] template retrieval source: do you show me doing [exercise: backflip] same related same/similar different target: can you show me how to do [exercise: yoga] paraphrase retrieval source: what can be seen inside [object: the basket] same same different same target: what does [object: the basket] mainly contain ## related work multilingual sentence encoders are widely studied and applied to downstream tasks in recent years. self-attention networks based multilingual sentence encoders, such as mbert @xcite and xlm-r @xcite , are pretrained on multilingual corpora in over 100 languages. labse @xcite ) encodes text to multilingual sentence embedding by training with 100 million sentence pairs in bilingual corpora of 109+ languages. @xcite proposes a centered embedding method that subtracts the mean embedding for each language from the sentence embedding, as well as a projection embedding method that projects bilingual using a parallel corpus. muse @xcite applies a translation based ranking task to one-billion weblab qa pairs to obtain a multilingual universal encoder. multilingual sbert @xcite ) extends pre-trained monolingual sbert @xcite to the multi-lingual version by mapping translations and original utterances into the same space. beyond atomic encoding, some research also focus on disentangling language specific and language-agnostic embeddings. @xcite learns to disentangle language syntax and semantic information by using aligned paraphrase data to train semantic and use word-order information to train syntax. bgt @xcite utilizes a deep variational probabilistic model together with transformers to learn better semantic embeddings in a bi-lingual setting by excluding language-specific information from the information shared across languages. @xcite proposes a method for distilling language-agnostic meaning embeddings by removing language-specific information from sentence embeddings generated by off-the-shelf multilingual sentence encoders. although these works extract both language-specific and language-agnostic embeddings, they are hard to support fine-grained semantic disentanglement. based on the work of tiyajamorn et al. (2021), we further extend the semantic extraction to a finegrained level. specifically, our proposed method fmda is able to extract not only the semantic information of the whole utterance but also part of it, i.e. template/carrier phrase semantic information by learning to disentangle language information at different levels. ## method this section describes the details of our proposed fine-grained multilingual disentangled autoencoder (fmda). figure 1(a) demonstrates the overview of the method. fmda is trained to extract language-specific non-semantic embedding, template semantic embedding, and residual-semantic embedding from a pmlm with three objectives: language identification, embedding reconstruction, and contrastive learning. figure 1(b) presents the objectives of the embedding reconstruction. figure 1(c ) shows the detailed contrastive learining design, in which fmda tries to minimize embedding difference between two paired utterances in different languages and maximize the embedding difference between two non-related utterances in the same language. each training step takes four utterances as inputs: utterance in language 1, utterance with same-semantic in language 2, and negative utterances with different semantic in language 1 and 2. ## conclusion in this paper, we introduced fmda, a lightweight encoding architecture that is able to disentangle fine-grained semantic information from languagespecific information in a multilingual setting. compared with previous works, the fmda distils 1) language embedding emb lang to encode the languagespecific information, 2) template embedding emb tp to encode the the backbone template of the sentence, and 3) the residual embedding emb rs to encode the residual information such as slot. such fine-grained representations allow retrieval applications at different levels under the nlu setting. two retrieval tasks conducted on the mas-sive dataset demonstrate that fmda's meaning embedding achieves the best performance on the cross-lingual semantic retrieval task and fmda's template embedding achieves the best performance on the mono-lingual template retrieval task. both constantly outperform the sota language-disentanglement baseline across multiple languages.