A Conditional Generative Matching Model for Multi-lingual Reply Suggestion (2021)

Abstract: We study the problem of multilingual automated reply suggestions (RS) model serving many languages simultaneously. Multilingual models are often challenged by model capacity and severe data distribution skew across languages. While prior works largely focus on monolingual models, we propose Conditional Generative Matching models (CGM), optimized within a Variational Autoencoder framework to address challenges arising from multilingual RS. CGM does so with expressive message conditional priors, mixture densities to enhance multilingual data representation, latent alignment for language discrimination, and effective variational optimization techniques for training multilingual RS. The enhancements result in performance that exceed competitive baselines in relevance (ROUGE score) by more than 10% on average, and 16%for low resource languages. CGM also shows remarkable improvements in diversity (80%) illustrating its expressiveness in representation of multi-lingual data.

## introduction automated reply suggestion (rs) helps users quickly process email and chats, in popular applications like gmail, outlook, microsoft teams, and facebook messenger, by selecting a relevant reply generated by the system, without having to type in the response. most existing rs systems are english mono-lingual models @xcite @xcite . we study the problem of creating multilingual rs models serving many languages simultaneously. compared to mono-lingual models, a universal multilingual model offers several interesting research questions and practical advantages. universal models can save compute resources and maintenance overhead for commercial systems supporting many regions. in addition it can benefit languages with insufficient data by informa-tion sharing from high resource languages and thus enhance experiences for users especially in lowlanguage resource regions. we investigate if a single multilingual rs model can replace multiple mono-lingual models with better performance, while overcoming the challenges in model capacity, data skew, and training complexities. trivially extending existing mono-lingual rs models to the multilingual setting (e.g. by jointly training with pre-trained multi-lingual encoders) tends to be sub-optimal, as multilingual models suffer from capacity dilution issue @xcite , where it improves performance on low resource languages while hurting the high resource ones. this arises, not only due to the severe data imbalance and distribution skew across languages, but also due to insufficient capacity and lack of inductive biases in models to represent the multi-modal distribution of languages. we postulate that deep generative latent variable models with variational auto-encoders (vae) @xcite are better suited to model the complex distribution of multi-lingual data, and be more data efficient for low resource languages. to this end, we propose the conditional generative matching model (cgm), a vae based retrieval architecture for rs to solve the above challenges. cgm enhances multilingual representation through: 1) expressive message conditional priors, 2) multi-component mixture density to represent different modalities of languages, and 3) alignment of latent components for language discrimination. in addition cgm incorporates training optimizations in the form of 1) loss regularizer, 2) learnable weights for loss components, 3) multi-sample loss estimation with variance scaling, and 4) focal loss, all of which lead to balanced representation and smooth convergence, a key challenge for variational training in multilingual settings. we conducted extensive ablation studies and comparisons with two competitive baselines to , where the encoders outputting θ m , θ r are considered extraneous to the generative model. the generative process is in the continuous space, with θ r generated conditioned on the input θ m and a gaussian prior z. the figures show three variations of this generative process. in prior work mcvae, z is sampled independently, while in cgm, it is conditional on θ m . cgm-m extends the message conditional prior with a gaussian mixture prior z c and a categorical prior c. ## background and preliminaries while rs has been modeled as a sequence to sequence model @xcite , it more commonly appears as an information retrieval (ir) system by ranking responses from a fixed set @xcite @xcite @xcite due to better control over quality and relevance for practical systems. we briefly describe two retrieval architectures from prior literature which serves as the baselines for our multilingual rs model. matching model @xcite consists of two parallel encoders [f ϕ m , f ϕ r ] to encode message and reply (m-r) pairs into a common encoding space, [θ m , θ r ] and trained to maximize a normalized dot product @xmath0. during prediction, the model finds the nearest neighbors of θ m with precomputed encodings from a fixed response set r @xcite . a language model bias is typically added to promote more common responses. the matching architecture is summarized as: matching conditional vae (mcvae) @xcite induces a deep generative latent variable model on the matching architecture, where a candidate response encoding is generated with θ @xmath1, z) conditioned on a latent prior z ∼ n (0, i). the generated θ r is used to match an actual response vector θ r from the fixed response set. the generative model of mcvae is shown in figure 1a . in mcvae, the encoders [f ϕ m , f ϕ r ] are pretrained using the matching formulation and kept frozen during the training. for prediction, mcvae samples response vectors from g w followed by scoring (eq 2) and a voting technique to rank replies over a fixed response set. mc-vae is trained in the variational framework by minimizing the negative evidence lower-bound (elbo) in equation 3 with a gaussian posterior q φ (mean and co-variance parameterized from (θ m , θ r )) and the reconstruction loss l m defined by eq. ( 1 ). we extend the matching and mcvae models to a multi-lingual setting by using pretrained multilingual bert (mbert) @xcite for [f ϕ m , f ϕ r ] similar to @xcite and jointly training the models for all languages. ## cgm: a conditional generative matching model for reply suggestion our initial analysis with universal models (jointly training models with all languages), reveals that the universal mcvae performs better than matching. however, simply training models jointly is not sufficient to achieve a models with high performance. first, the highly imbalanced nature of multi-lingual data leads to over-or under-fitting across languages resulting in performance worse than separately trained mono-lingual models. second, training multi-lingual mcvae proved is due to the reliance on a pretrained matching model: it is not clear how to find a suitable matching model checkpoint for initializing the mcvae. finally, since the text encoders for mcvae are frozen during training, there is limited cross lingual transfer and improvement for low resource languages. unfreezing the layers led to divergence of the model. to address the limitations of mcvae, we propose an enhanced conditional generative matching (cgm) model, for the retrieval based rs with inductive biases for the multi-lingual data and effective training techniques for creating high quality universal models. ## related work vaes have been used in retrieval based q&a @xcite , document matching @xcite , and recommendations (chen and de rijke, 2018). cgm for rs is most closely related to @xcite but differs in the expressive conditional priors, multi-component mixture density priors, language alignment, and training optimizations which makes it effective in a multi-lingual setting. for multi-task scenarios, vaes can offer significant modeling efficiencies @xcite with additional improvements through mixture model priors, e.g. in @xcite for unsupervised clustering, in @xcite for unsupervised meta-learning, and in @xcite as a multi-modal variational mixture-of-experts. vaes can also improve multilingual representation for low resource languages, e.g. in models like bert @xcite , in @xcite for document classification, in @xcite for disentangling phonemes for speech synthesis, and in @xcite for neural machine translation. vaes can improve diversity in language generation and retrieval tasks @xcite @xcite through better modeling efficiencies. such results motivated us to apply vaes for multilingual rs. we may also consider alternative to vaes such as training auxiliary tasks with adapters @xcite , adversarial learning @xcite @xcite , and mixing pre-training and fine-tuning @xcite to improve modeling in multilingual setting. this is subject of future work. we also plan to experiment with higher capacity multilingual encoders such xlm-r @xcite and infoxlm @xcite to further improve the performance. however, the choice of the base encoder is orthogonal to the improvements (especially on diversification) shown in this paper. as noted in prior work, multilingual training can have capacity dilution issues @xcite @xcite . overall, multilingual models are closing the gap with monolingual counterparts for wide range of tasks @xcite @xcite , and as shown in this paper, even surpass them. careful sampling strategies, and techniques such as translation language model (tlm) can alleviate the "curse of multilinguality" (lample and conneau, 2019) but we show improvements without additional data augmentation (translation pairs), and with simple uniform sampling. ## conclusions in this paper we present a conditional generative matching model (cgm) for retrieval based suggested replies. cgm not only provides relevance gains (15%), but also substantial improvements in diversity (80%). while cgm clearly advances the state of art for modeling multi-lingual rs systems, it also illustrates that through proper model choices and training optimizations, we can surpass and replace monolingual models. this is important for both industry and academia and suggests similar strategies to be applied across diverse tasks. this is subject of future work. wenhao yu, lingfei wu, qingkai zeng, shu tao, yu deng, and meng jiang. 2020. crossing variational autoencoders for answer retrieval. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 5635-5641, online. association for computational linguistics. biao zhang, deyi xiong, jinsong su, hong duan, and min zhang. 2016. variational neural machine translation. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 521-530. mozhi zhang, wei wang, budhaditya deb, guoqing zheng, milad shokouhi, and ahmed hassan awadallah. 2021. a dataset and baselines for multilingual reply suggestion. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 1207-1220, online. association for computational linguistics. tiancheng zhao, ran zhao, and maxine eskénazi. 2017. learning discourse-level diversity for neural dialog models using conditional variational autoencoders. in acl. xiangyang zhou, daxiang dong, hua wu, shiqi zhao, dianhai yu, hao tian, xuan liu, and rui yan. 2016. multi-view response selection for human-computer conversation. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 372-381, austin, texas. association for computational linguistics. xiangyang zhou, lu li, daxiang dong, yi liu, ying chen, wayne xin zhao, dianhai yu, and hua wu. 2018. multi-turn response selection for chatbots with deep attention matching network. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1118-1127, melbourne, australia. association for computational linguistics.