Language Scaling for Universal Suggested Replies Model (2021)

Abstract: We consider the problem of scaling automated suggested replies for a commercial email application to multiple languages. Faced with increased compute requirements and low language resources for language expansion, we build a single universal model for improving the quality and reducing run-time costs of our production system. However, restricted data movement across regional centers prevents joint training across languages. To this end, we propose a multi-lingual multi-task continual learning framework, with auxiliary tasks and language adapters to train universal language representation across regions. The experimental results show positive cross-lingual transfer across languages while reducing catastrophic forgetting across regions. Our online results on real user traffic show significant CTR and Char-saved gain as well as 65% training cost reduction compared with per-language models. As a consequence, we have scaled the feature in multiple languages including low-resource markets.

## introduction automated suggested replies or smart replies (sr) assist users to quickly respond with a short, generic, and relevant response, without users having to type in the reply. sr is an increasingly popular feature in many commercial applications such as gmail, outlook, skype, facebook messenger, microsoft teams, and uber @xcite @xcite @xcite . while the initial versions of this feature mostly targeted english users, making it available in multiple languages and markets is important not only from the perspective of product expansion but also from a linguistic inclusivity point of view. in this paper we consider the problem of rapid scaling of the sr feature to multiple languages for outlook. to develop such a system at production scale, we are faced with the following challenges. to reduce the cost of model management, we propose to build a single universal sr model, capable of serving multiple languages and markets. to overcome data constraints, we propose to use augmentation with machine-translated (mt) data for languages without supervised data. to overcome privacy constraints, we propose a continual learning framework, where the model is trained sequentially across regions. to alleviate catastrophic forgetting @xcite in the continual learning process, we reinforce the universal properties via multi-task learning approach with public task-agnostic data, and an adapter-based model architecture that leverages domain-specific sr data and mt data. our experimental results followed with improvements shown on real user traffic illustrate the ef-fectiveness of the approach. as a consequence, we have rapidly scaled the feature in several languages including low-resource markets. multilingual training for universal models is often very tricky to work in practice (especially with our data constraints). thus, we demonstrate a significant accomplishment of a multi-lingual sr system running at production scale on millions of users, which saves resources while improving performance. ## core sr model the sr feature is similar to open-domain chatbots and task-oriented conversational agents, @xcite @xcite @xcite . in terms of usage, sr is closer to the latter, in that it assists users to complete a reply, instead of continuing an openended dialog. following commonly used ir-based models in commercial sr applications @xcite , we use a dual encoder matching model for our sr system. the matching model has two parallel encoders projecting input message and corresponding reply into a common representation space. different encoders such as feed-forward and bilstm layers can be used here @xcite . more recently, @xcite @xcite show considerable improvements with transformer-based pre-trained models. our english sr model uses a bert equivalent @xcite encoder, while our mono-lingual baselines in other languages use bilstm encoders. the model is trained on one-on-one messagereply (m-r) pairs from commercial email data. we minimize the symmetric loss function. it is a modified softmax on dot products between m-r encoding in equation 1 where s i,@xmath0. as described in @xcite , it was shown to improve the relevance by targeting at bi-directional conversational constraints. ir-based model requires a fixed response set. to generate that, we collect differentially private (dp) @xcite and anonymized replies, filtered for sensitive content from the training data which preserves user privacy while mining actual user responses. furthermore, we use human curation to edit responses for cultural-sensitivity, genderneutrality, etc. dp filtration requires a large amount of data due to low yields. for low-resource markets, we translate english responses with human curation for cultural adaptation to languages and locales. during prediction, we compute the matching score (•) between the message and pre-computed response set vectors. similar to @xcite , we add a language-model (lm) penalty representing the popularity of responses to bias the predictions towards more common ones. translated responses inherit the penalty score from the corresponding english responses. using this score in equation 2 we first select top n 1 responses, and down-select to top n 2 after deduplication using lexical clustering, before presenting to users. the universal sr model consists of parallel encoder architecture trained using symmetric loss function similar to the core sr model. we initialize the m-r encoders with infoxlm @xcite , an xlm-roberta @xcite equivalent multi-lingual model as shown in as figure 1 (a) which creates language-agnostic text representation across 100 languages. the encoder is pre-trained with both publicly available and internal proprietary corpora and has shown good cross-lingual transfer capabilities on benchmarks such as xnli @xcite . using a universal pre-trained model in itself enables language expansion. however, as we discuss next, data movement constraints made training the universal model tricky, with performance frequently worse than single mono-lingual models. ## experiments and results the training data is collected and processed without any eyes access from commercial users in outlook email system. to be more specific, we filter 50m m-r pairs from one-to-one conversations for each high-resource language, and translate 20m m-r pairs for each low-resource language. considering the m-r length distribution, we truncate m-r pairs to (96, 64) tokens as training data, and filter out messages longer than 96 tokens during inference, so that the model is more focused on providing quick responses to short messages. the response set size for each language is 20k, filtered or trans-created from english native data. in all three stages of training, we use an effective batch size of 16k. we utilize the adam optimizer (kingma and ba, 2014) with weight decay and set peak learning rates as [5e-4, 3e-4, 1e-4] for three stages respectively. we train up to 30 epochs from which the best model is selected based on validation set loss over all languages. for mlm/tlm objectives, we use single-token masking, the task proportion is set as 0.5. the final loss of the model is sum of symmetric loss and auxiliary task loss. for adapters, we use the hidden dimension of 256 in the bottleneck architecture and initialize these parameters with a normal distribution of mean 0 and standard deviation 0.01. according to our observation, high standard deviation for initialization can cause divergence. all experiments are conducted with 16 nvidia v100-32gb gpu cards. during prediction, we pick top n 1 = 30 responses according to equation 2, and then cluster the ranked results and down-select n 2 = 3 responses as final prediction. ## conclusions this paper presents our approach of scaling automated suggested replies with one universal model. faced with compute resource and data privacy constraints, we propose a multi-task continual learning framework with auxiliary tasks, and data augmentation with adapter-based model architecture. the universal model in production saves significant compute resources and model management overhead, while allowing us to train across regional data boundaries. in addition, the process allows us to cold-start in new markets even when no supervised data exists. based on the promising offline and online results, we have deployed the model in several languages and plan to extend the process for 20 languages around the world.