End-to-End Autoregressive Retrieval via Bootstrapping for Smart Reply Systems (2023)

Abstract: Reply suggestion systems represent a staple component of many instant messaging and email systems. However, the requirement to produce sets of replies, rather than individual replies, makes the task poorly suited for out-of-the-box retrieval architectures, which only consider individual message-reply similarity. As a result, these system often rely on additional post-processing modules to diversify the outputs. However, these approaches are ultimately bottlenecked by the performance of the initial retriever, which in practice struggles to present a sufficiently diverse range of options to the downstream diversification module, leading to the suggestions being less relevant to the user. In this paper, we consider a novel approach that radically simplifies this pipeline through an autoregressive text-to-text retrieval model, that learns the smart reply task end-to-end from a dataset of (message, reply set) pairs obtained via bootstrapping. Empirical results show this method consistently outperforms a range of state-of-the-art baselines across three datasets, corresponding to a 5.1%-17.9% improvement in relevance, and a 0.5%-63.1% improvement in diversity compared to the best baseline approach. We make our code publicly available.

## introduction reply suggestion, or smart reply (sr), systems are a staple component of many commercial applications such as gmail, skype, outlook, microsoft teams, linkedin and facebook messenger. they help the user process chats and emails quicker by offering a set of canned replies which can be clicked without requiring manual typing. however, dialogue is known to be a one-to-many problem @xcite ) -namely, for any given message, there are multiple possible replies. to reflect this uncertainty, systems should present a diverse set of options to the user. for instance, given the message how are you?, an sr system could suggest: {i'm good; ok; not great}. resultantly, the quality of a given reply depends not only on the message, but on the other replies in the reply set. several prior works explore solutions to this problem such as removing near duplicates, penalising inter-reply similarity @xcite , clustering by intent @xcite , learning latent variables @xcite , or model-based simulation @xcite . however, these methods share a common design choice (figure 1a ): (1) a retrievalbased matching model, which has learned a shared embedding space between messages and replies, returns a shortlist of top scoring replies; (2) this shortlist is refined through some diversification procedure to obtain the final reply set. unfortunately, this assumes that the initial shortlist contains at least one good reply set. in practice, we find matching models often search myopically, only retrieving candidates that are very similar to one another (figure 1a ). thus, the chosen reply set often fails to reflect a diverse range of user intents, while latency constraints make more sophisticated diversification techniques or larger shortlists prohibitive @xcite . an intuitive, but -to the best of our knowledge -unexplored, solution to this problem is to conduct the retrieval autoregressively, with each reply conditioned on both the initial message and the previous replies in the set. unfortunately, this approach encounters a second problem, namely, the lack of any datasets containing (message, reply set) pairs @xcite . in practice, sr systems are trained on individual (message, reply) pairs obtained from conversation datasets, while the task of presenting multiple diverse replies to the user is outsourced to a separate diversification module. figure 1 : previous methods [a] compared to our approach, star @xcite . the example displayed is taken from the dailydialog test set, and compares the predictions of star with simsr @xcite , the next best method. our method's suggestions present a diverse range of topics/intents to drive the conversation. to meet this dual need, we present both (i) a bootstrapping method for creating a high-quality dataset of (message, reply sets) and (ii) a novel autoregressive retrieval model which predicts sequences of replies. for solving (i), we observe how model-based planning algorithms have been known to serve as a powerful policy improvement operator @xcite , including in several nlp systems @xcite . specifically, the outputs of a model-based planning algorithm can be used to bootstrap a sr system. further, by conducting this planning offline we are able to leverage two key advantages: (1) the system is free of the latency constraints of online inference, and therefore can increase the search space coverage of the planning algorithm; (2) the system can leverage information that would not be available during inference, such as the ground-truth reply, to further guide the search process. for (ii) we unify both steps of the standard sr pipeline into a single end-to-end model, which mitigates the myopic search, and allows the model to learn to diversify its predictions in a principled way through gradient-based learning. to this end, we present star (suggested replies with t5 and autoregressive retrieval) (figure 1b ). at a high level, star is a text-to-text model trained to output sequences of replies, where each reply is conditioned both on the initial message and the previous replies in the sequence. concretely, we instantiate our method with the t5 pretrained model @xcite . we expand t5's vocabulary by treating each reply in the candidate pool as a novel token, and demonstrate a simple-yet-effective technique for initialising the new token embeddings, which leverages the model's existing semantic priors. notably, by treating each reply as a token, we limit the number of autoregressive decoding steps required, keeping the model's efficiency comparable to other retrieval-based methods. empirically, we evaluate our approach on three benchmarks: reddit @xcite , which is the only publicly-available sr benchmark, as well as personachat @xcite and dailydialog @xcite which are both widely-used in dialogue research more broadly @xcite , and share a similar conversational style with sr apps. we demonstrate superior performance over state-ofthe-art baselines across all datasets, corresponding to a 5.1%-17.9% improvement in relevance, and a 0.5%-63.1% improvement in diversity compared to the best baseline approach. we further show comparable efficiency to previous methods, and perform a range of ablations to motivate our design choices. in summary, our key contributions are as follows: ## related work smart reply the proprietary nature of data from email and chat applications has led several previous works to use publicly-available dialogue datasets @xcite @xcite to benchmark sr methods, due to their analogous conversational nature. while early sr systems used generative models @xcite , current production systems favour retrieval methods due to their greater controllability of outputs and superior latency @xcite . increasing the diversity of reply suggestions is a key focus of previous work, which has been attempted by: (1) mapping replies to discrete intents / topics @xcite @xcite ; (2) re-weighting replies according to their similarity with other replies in the set @xcite ; (3) learning continuous latent variables to generate multiple queries @xcite ; (4) using model-based simulation to iteratively search and evaluate the relevance of candidate reply sets @xcite . our proposed method differs from all of these approaches in that our model learns to account for the interdependencies between replies through end-to-end backpropagation. autoregressive retrieval integrating neural retrieval into the well-established paradigm of textto-text models is of growing interest. earlier work focuses on outputting a document id given a query @xcite . further work has extended this by considering alternate ways of representing the document ids, such as through unique substrings @xcite . another line of work has used autoregressive retrieval for the entity linking task @xcite . there, the motivation is to reduce the large number of entities by relying on the text-to-text model's pre-existing vocabulary, rather than having to retrieve embeddings from a memory-intensive dense index. our proposed method differs considerably from these previous works both in instantiation and motivation. instantiation-wise, we generate multiple repliescritical to making this possible is the novel bootstrapping technique for creating the dataset of (message, reply set) pairs to train on. motivation-wise, our goal is to be able to condition each reply on both the input message and previous replies in the set, enabling the model to learn to predict se-quences of replies in a differentiable way. bootstrapping the idea of bootstrapping training data from limited resources has received significant recent interest in nlp, given the newly demonstrated few / zero-shot capabilities of many large language models @xcite . it has seen usage in few-shot shot text-classification @xcite , semantic similarity @xcite , tool-usage @xcite , retrieval @xcite , sequence generation @xcite , and instructiontuning @xcite @xcite , amongst others. these techniques can also be seen as a form of knowledge distillation @xcite , except that the training typically involves predicting the exact token targets, rather than using the soft probabilities of a teacher model. although sometimes these techniques are used as an addition to supervised learning @xcite , in our case there are no datasets containing the ideal reply sets to suggest to the user. instead, we must bootstrap this in a more unsupervised way, by transforming a dataset of (message, reply) pairs into a dataset of (message, reply set) pairs. ## methodology in this section, we first describe the model-based planning process used to obtain the bootstrapped dataset of (message, reply set) pairs (section 3.1). then, we show how the star architecture can be trained on this dataset (section 3.2). ## experimental results we focus our efforts on answering the following research questions: (rq 1 ) how does star com-pare to existing state-of-the-art methods? (section 5.1, 5.4); (rq 2 ) which components of the data collection algorithm and fine-tuning have the largest impact on star's performance? (section 5.2); (rq 3 ) how efficient is star in inference? (section 5.3) ## conclusion we introduce star, an autoregressive retrieval system for sr, which is an end-to-end text-to-text model that sequentially predicts replies conditioned on an initial message. to train star, we demonstrate an approach to bootstrap a dataset of highquality (message, reply set) pairs, from regular dialogue datasets containing only (message, reply) pairs. empirically, our results show significant improvement over existing state-of-the-art sr baselines, across multiple datasets, corresponding to a 5.1%-17.9% improvement in relevance, and a 0.5%-63.1% improvement in diversity compared to the best baseline approach. future work could extend these techniques to other set-prediction tasks: e.g., in ir the relevance of each document depends on the quantity of new information it contains compared to other documents in the set. in recommender systems, use cases include: tailoring a user's news feed requires that the news articles presented are not simply duplicates of the same story; designing a bespoke music playlist requires songs to be unified by common themes but also sufficiently distinct from one another to maintain the listener's interest. other lines of future work include considering alternate strategies for initialising the reply embeddings, beyond the bag-of-words initialisation demonstrated in this paper.