An Efficient Conversational Smart Compose System (2023)

Abstract: Online conversation is a ubiquitous way to share information and connect everyone but repetitive idiomatic text typing takes users a lot of time. This paper demonstrates a simple yet effective cloud based smart compose system to improve human-to-human conversation efficiency. Heuristics from different perspectives are designed to achieve the best trade-off between quality and latency. From the modeling side, the decoder-only model exploited the previous turns of conversational history in a computation lightweight manner. Besides, a novel phrase tokenizer is proposed to reduce latency without losing the composing quality further. Additionally, the caching mechanism is applied to the serving framework. The demo video of the system is available at https://youtu.be/U1KXkaqr60g.We open-sourced our phrase tokenizer in https://github.com/tensorflow/text .

## introduction online conversations are happening in every corner of the world in every second. people relies on different channels like daily chatting apps, e.g. messages, whatsapp, or online customer service to communicate with friends, families, colleagues and even acquaintance. within conversational applications, efficient and smart assistant functions for users are long-desired. smart compose @xcite is a well-known smart writing feature that actively predicts the next couple of words in realtime to assist human writing. it saves user's time by cutting back on repetitive idiomatic writing. in this paper, we build a smart compose system for chatting applications to improve user's communication efficiency and enhance user experience. the outlook of the demo is shown in fig 1 . we assume two or multiple users are discussing some random topic in a chat room. and smart compose is working under the hood to suggest what to type next. the full demo is in https://youtu.be/u1kxkaqr60g . to the best of our knowledge, although smart compose is a well-known application, it has not been well-designed for conversation scenarios. one basic requirement of smart compose is the low inference latency. if the next words are not shown up in less than 100ms, users will likely type them themselves. and there are two challenges out of this. the first challenge is the conflict between latency and long conversation context. during the human-to-human conversation within the chatting applications, there are multiple turns of conversation from multiple users, making the conversation history informative. attention over the whole conversation history would bring too much latency. however, simply ignoring the conversation history or using an average embedding of history to represent previous context @xcite is not exploiting the rich information within the conversational flow between users. therefore how to effectively and efficiently extract information from conversation history is the key consideration for model designing. the second challenge comes from the fact that users would prefer a longer sug-gestion for conversation but generating a longer sequence is likely to involve more latency. in the auto-regressive generative model for example, the decoding time is linearly growing with the decoding steps. there is no clear solution on how to avoid the extra latency on the longer suggestion. in this paper, we proposed a new solution to address above challenges and built an end-to-end cloud based smart compose system for human-tohuman chatting applications. our smart compose system can achieve high exact match rate with low latency (less then 100ms) under acceptable suggestion length. our contributions are three-fold. ## related work one of the most successful applications for smart compose is gmail @xcite , where the title of the email, the previous paragraphs, and the previous text of the current typing are combined to predict the next words. following this success, similar models have also been used in other applications like document editing (e.g., google docs) or input method (e.g., gboard). although this feature is not new, previous work mainly focuses on general writing assist and not is specially designed for conversation scenarios. approach wise, there are two major categories. the first category is the dual encoder based model @xcite @xcite which exploited contrastive learning: the predefined whitelist is built in advance, and the prefix tries to match the most possible one in the list. this kind of solution is also widely used in practical retrieval or recommendation systems. although the dual encoder is extremely fast, the predefined whitelist limits the headroom for model quality from smart compose. the second category is language modeling, a fundamental and indispensable component of many natural language processing. recently it is becoming even more powerful with a larger model size @xcite @xcite . however, with the latency requirement, models should be designed smaller and more light-weighted for the smart compose @xcite @xcite . compared to dual-encoder, it removes the dependency on the whitelist and could achieve better matching accuracy. however, as it is involved with the autoregressive decoding process, the latency is generally higher and positive relative to the number of decoding steps. although non-autoregressive based approaches @xcite could potentially reduce the latency for text generation, it has been well-known to suffer from quality regression. our smart compose system is more conversational oriented and can achieve the best trade-off between quality and latency. our model share the benefit of higher quality with the language modeling, but keep the latency low with proposed techniques. ## model design in this section, we will outline the modeling design. firstly, we will first introduce the conversational inputs for the model and what features we are extracting from the conversation history. then we will go through the model architecture used to take the conversational input and output the predicted suggestions. finally, we will discuss how the loss is calculated during training and some implementation details. ## phrase tokenizer autoregressive decoding is the primary way of generating text with a transformer decoder. one drawback of this mechanism is the latency issue: with n output units, the model will forward pass n times in a sequential manner. in this paper, we design a token that is beyond a single word (a phrase with ngram) and a tokenizer that emits a group of units at a time, with the fallback mechanism to emit a single unit so that we can decrease the n (i.e., number of output tokens) here to reduce the latency. note that the basic element of the phrase is a word following the findings of (chen et al., 2019) that the wordbased vocabulary can achieve better performance for smart compose. nevertheless, our approach can also be applied to word-piece or sentence-piece models. ## serving system design we will discuss the serving system design and heuristics in this section. specifically, we will introduce how we design the inference graph to take advantage of the caching of conversation history. besides, we will illustrate the server-side components of the system. ## experiments 6.1 data and model setup our experimental data is based on conversational data collected in existing conversational apps. we use the previous 10 conversation as conversation history and restrict the max sequence length as 256 tokens. for the model configuration, we used the the transformer with 8 attention heads. the input dimension as well as model dimension is set to 256 and the dimension of feed-forward network is set to 1024. we did not find dropout useful, so the models are trained without dropout. all the models for experiments presented in this paper are trained on 8x8 dragonfish tpu with a per-core batch size of 32 (effective batch size of 4096). the learning rate schedule is ramped up linearly from 0 to 8.0e-4 during the first 10,000 steps, and then it decays exponentially to zero till 500,000 steps. we use cpu for serving. the total model size is around 5mb.