Model-Based Simulation for Optimising Smart Reply (2023)

Abstract: Smart Reply (SR) systems present a user with a set of replies, of which one can be selected in place of having to type out a response. To perform well at this task, a system should be able to effectively present the user with a diverse set of options, to maximise the chance that at least one of them conveys the user’s desired response. This is a significant challenge, due to the lack of datasets containing sets of responses to learn from. Resultantly, previous work has focused largely on post-hoc diversification, rather than explicitly learning to predict sets of responses. Motivated by this problem, we present a novel method SimSR, that employs model-based simulation to discover high-value response sets, through simulating possible user responses with a learned world model. Unlike previous approaches, this allows our method to directly optimise the end-goal of SR–maximising the relevance of at least one of the predicted replies. Empirically on two public datasets, when compared to SoTA baselines, our method achieves up to 21% and 18% improvement in ROUGE score and Self-ROUGE score respectively.

## introduction automated response suggestion, or smart reply (sr), is rapidly becoming a staple feature of many email and chat systems such as gmail, skype, outlook, microsoft teams, linkedin and facebook messenger. given a message, sr systems present the user with a selection of possible responses, e.g. how are you? → {i'm good; i'm ok; not great}, which they can click in place of having to type out a reply. with the growth of communication over smaller devices that are poorly suited for manual typing @xcite , such as smartphones and smart watches, sr is becoming an increasingly more important feature. while early methods in sr incorporated sequence-to-sequence models @xcite , the current mainstream approach favours matching models which separately encode the message and reply into a shared latent space and retrieve the nearest neighbour response @xcite @xcite . this has advantages in a production context, as it enables the model to retrieve replies from a fixed response set, maintaining greater controllability of model outputs; further, the latent representations for the response set can be pre-computed prior to inference, enabling faster latency. however, the naive approach of simply retrieving top-k highest-scoring candidates from the matching model often fails to produce a sufficiently diverse set of reply options. for instance, in response to the message how are you?, if the first predicted response is i'm good, predicting i'm doing well as the second response provides limited incremental value, as it carries equivalent semantic meaning. by contrast, not great would be more useful, as it captures an alternative semantic meaning a user might wish to convey. in summary, one must account for the interdependencies between replies. previous methods have sought to implicitly account for these interdependencies such as through clustering by intent/topic, learning latent variables or re-scoring replies to include inter-reply similarity @xcite @xcite . however, these techniques face two limitations: (1) they require hard-coded trade-offs between message-reply relevance and inter-reply diversity; (2) jointly optimising these two metrics is only partially correlated with the end goal of sr-maximising the relevance at least one of the predictions. ideally, it would be more principled if the model could simply optimise over this end goal. in so doing, we hypothesise performance would improve, while a good amount of diversity should also naturally emerge, insofar as it is correlated with performance on the task. however, directly optimising this metric figure 1 : overview of our approach. we combine a retrieval stage, which obtains the initial reply shortlist y n , followed by a simulation stage, which iteratively searches for reply sets y k from that shortlist, and evaluates their relevance against a set of simulated replies y m . concretely, our method-simsr (figure 1 )comprises an initial retrieval stage, followed by an iterative simulation stage. we first retrieve a shortlist of replies from a larger candidate pool, using a learned neural matching model, conditioned on a given message. in parallel, we also retrieve a number of simulated replies using the same method. next, for the simulation stage, we use a search module to select a reply set comprising three responses from the shortlist. then, we use a valuation module, which computes the expected similarity between the simulated replies and the most similar response from the reply set. this can be computed through a simple marginalisation process, using the probabilities and corresponding simulated replies provided by the world model. this process of search and valuation is iterated until the search algorithm terminates, and finally returns the highest scoring reply set. quantitatively, our experiments show consistent out-performance against existing sota methods across two relevant datasets-reddit and persona-chat-achieving up to 21% and 18% improvement in rouge score and self-rouge score respectively. simsr also runs at a comparable speed to other methods, because the simulation is highly parallelisable and the matching model only needs to encode the message once for both its initial retrieval and world model roles. in summary, our key contributions are: we make our code available for reproducibility. 1 2 related work smart reply. in industry, sr has a range of applications from email systems to instant messaging. naturally, the data from these is not publicly available to train on. instead, recent work has made use of publicly available dialogue datasets such as reddit @xcite , which is sufficiently similar given sr applications are principally concerned with dialogue. while the earliest sr systems used sequence-to-sequence models @xcite , nowadays retrieval methods prevail which select a response from a pre-defined pool of candidates @xcite , i.e. matching models. by itself however, the matching model has no way to ensure that the chosen reply set is sufficiently diverse. one approach to this is to ensure that no two responses in the reply set share the same topic/intent @xcite @xcite . however, this becomes more difficult in an open-domain setting, where the range of topics/intents is difficult to pre-define. as a result, other approaches have focused on more finegrained diversification through conditional variational autoencoder techniques, which learn topics/intents across a continuous latent space during training @xcite . maximum marginal relevance, which re-weights responses according to how similar they are with one another, has also been shown to work well @xcite . our method differs from these approaches in that they employ diversity in a post-hoc manner which does not directly optimise the end goal of sr-maximising the relevance of at least one of the predicted replies. ## simsr for any given message x, there is uncertainty about the response y, which we assume to be sampled from some distribution y . this is commonly referred to as the one-to-many problem @xcite and is due to several reasons, such as unknown facts about the user and their intent. for example, the reply to can you meet for coffee at 2pm? is likely to be conditioned on factors such as the user's schedule or their interest in meeting, which is unknown to a vanilla sr system. as a result, matching models that simply select the most likely individual replies only achieve a lower bound of potential performance. this can be represented by the following inequality: to do this, we hypothesise it is important to develop a method that accounts for the interdependencies between replies, i.e. which can evaluate sets of replies, rather than only individually scoring replies. algorithm 1 and figure 1 overview our method, which can be applied directly during inference. the matching model first retrieves a shortlist of n replies from a pool of pre-computed candidates y r (section 4.1). then we combine a search module which selects and constructs reply tuples from this shortlist to evaluate (section 4.4) and a valuation module (section 4.3) which computes an expected score between a given reply set and a list of simulated replies (section 4.2). note that as our method does not require learning any new parameters, it can be applied to reply sets of arbitrary sizes during inference. ## experiments we now turn our attention towards empirical testing of simsr, addressing the following research questions: ## conclusion in this work, we have presented a method for generating sets of replies for smart reply systems, using model-based simulation and a range of search strategies to discover high-value reply sets, without the need for any additional training. our method outperforms existing sota methods on both datasets tested, and we have supported our results by detailed analysis of the effect of different search strategies, demonstration of the impor-