Improving Text Auto-Completion with Next Phrase Prediction (2021)

Abstract: Language models such as GPT-2 have performed well on constructing syntactically sound sentences for text auto-completion tasks. However, such models often require considerable training effort to adapt to specific writing domains (e.g., medical). In this paper, we propose an intermediate training strategy to enhance pre-trained language models’ performance in the text auto-completion task and fastly adapt them to specific domains. Our strategy includes a novel self-supervised training objective called Next Phrase Prediction (NPP), which encourages a language model to complete the partial query with enriched phrases and eventually improve the model’s text auto-completion performance. Preliminary experiments have shown that our approach is able to outperform the baselines in auto-completion for email and academic-writing domains.

## introduction natural language interface (nli) applications such as personal assistants (e.g., amazon alexa, apple siri, google assistant, and microsoft cortana) and search engines (e.g., google) have become an integral part of our everyday life. among the many features in nli applications, text auto-completion, which aims to suggest words, phrases, and sentences that complete the user's textual input, is a common, but key feature. smart reply @xcite and smart compose @xcite are two recent works that provide contextual assistance to aid users in completing everyday text such as emails, search engine inputs, etc. while recent advances in deep neural models have shown impressive performance on the text auto-completion task, these models generally require a large amount of everyday text and huge amount of computing power for training to generate adequate suggestions @xcite . the ## overview in this section, we first formalize the auto completion problem, and then introduce the workflow of our intermediate training strategy. ## next phrase prediction the key idea of the next phrase prediction (npp) objective is to train a text-to-text transformer to complete the partial query with adequate phrases. the underlying intuition of our proposed approach is as follows: (1) phrases tend to express meaning beyond simple word concatenation. for example, noun phrase such as "recurrent neural network" is constructed by three different words ("recurrent", "neural", "network"), where each word has its own meaning. (2) common phrases tend to be used on their own in the text. for instance, the prepositional phrase such as "in this paper" frequently appears in academic writing domain. unlike existing language models that are trained to neglect such characteristics of phrases and predict the next word or span of the text, text auto-completion can be improved by performing phrase-level text com-pletion as an intermediate training strategy in an effort to make the most of the phrase. specifically, npp involves two main steps: (i) phrase extraction, and (ii) generative question answering (qa). ## conclusion in this paper, we propose a novel intermediate training strategy that encourages the model to complete the partial query with enriched phrases and eventually improving the performance of the text auto-completion system. our proposed approach enhances state-of-the-art language model's performance by intermediately training it with our next phrase prediction self-supervised objective. preliminary experiments have shown that our approach is able to outperform the baselines in auto-completion for email and academic-writing domains with only around 1.2b tokens of training. for future work, we aim to experiment our proposed approach on text auto-completion in more writing domains and develop a demonstration system to better showcase our approach in text auto-completion.