C onve RT : Efficient and Accurate Conversational Representations from Transformers (2020)

Abstract: General-purpose pretrained sentence encoders such as BERT are not ideal for real-world conversational AI applications; they are computationally heavy, slow, and expensive to train. We propose ConveRT (Conversational Representations from Transformers), a pretraining framework for conversational tasks satisfying all the following requirements: it is effective, affordable, and quick to train. We pretrain using a retrieval-based response selection task, effectively leveraging quantization and subword-level parameterization in the dual encoder to build a lightweight memory- and energy-efficient model. We show that ConveRT achieves state-of-the-art performance across widely established response selection tasks. We also demonstrate that the use of extended dialog history as context yields further performance gains. Finally, we show that pretrained representations from the proposed encoder can be transferred to the intent classification task, yielding strong results across three diverse data sets. ConveRT trains substantially faster than standard sentence encoders or previous state-of-the-art dual encoders. With its reduced size and superior performance, we believe this model promises wider portability and scalability for Conversational AI applications.

## introduction dialog systems, also referred to as conversational systems or conversational agents, have found use in a wide range of applications. they assist users in accomplishing well-defined tasks such as finding and booking restaurants, hotels, and flights @xcite @xcite , with further use in tourist information @xcite , language learning @xcite , entertainment @xcite , and healthcare @xcite . they are also key components of intelligent virtual assistants such as siri, alexa, cortana, and google assistant. data-driven task-oriented dialog systems require domain-specific labelled data: annotations for intents, explicit dialog states, and mentioned entities @xcite @xcite @xcite . this makes the scaling and maintenance of such systems very challenging. transfer learning on top of pretrained models @xcite provides one avenue for reducing the amount of annotated data required to train models capable of generalization. pretrained models making use of languagemodel (lm) based learning objectives have become prevalent across the nlp research community. when it comes to dialog systems, response selection provides a more suitable pretraining task for learning representations that can encapsulate conversational cues. such models can be pretrained using large corpora of natural unlabelled conversational data @xcite . response selection is also directly applicable to retrieval-based dialog systems, a popular and elegant approach to framing dialog @xcite @xcite @xcite . 1 response selection is a task of selecting the most appropriate response given the dialog history @xcite @xcite @xcite . this task is central to retrieval-based dialog systems, which typically encode the context and a large collection of responses in a joint semantic space, and then retrieve the most relevant response by matching the query representation against the encodings of each candidate response. the key idea is to: 1) make use of large unlabelled conversational datasets (such as reddit conversational threads) to pretrain a neural model on the general-purpose response selection task; and then 2) fine-tune this model, potentially with additional network layers, using much smaller amounts of task-specific data. dual-encoder architectures pretrained on response selection have become increasingly popular in the dialog community @xcite @xcite . in recent work, @xcite show that standard pretraining lm-based architectures cannot match the performance of dual encoders when applied to dialog tasks such as response retrieval. ## methodology pretraining on reddit data. we assume working with english throughout the paper. simplifying the conversational learning task to response selection, we can relate target dialog tasks to general-domain conversational data such as reddit @xcite . this allows us to fine-tune the parameters of the task-specific response selection model, starting from the general-domain response selection model pretrained on reddit. similar to @xcite , we choose reddit for pretraining due to: 1) its organic conversational structure; and 2) its unmatched size, as the public repository of reddit data comprises 727m (input, response) pairs. 3 ## results and discussion model size, training time, cost. response selection on reddit. the results are summarized in we have also verified that the first context from the dialog history is most beneficial for performance of the multi-context convert variant with a simple experiment. when using only the first context from the dialog history, performance on reddit drops only slightly: from 71.8% (10 contexts from history) to 71.1% with the full model including the immediate preciding context, and from 34.0% (10 contexts) to 30.9% (1 context only) when we exclude the intermediate context (see §2.2 again). ## conclusion we have introduced convert, a new light-weight model of neural response selection for dialog, based on transformer-backed dual-encoder networks, and have demonstrated its state-of-the-art performance on an array of response selection tasks and in transfer learning for intent classification tasks. in addition to offering more accurate conversational pretraining models this work has also resulted in more compact conversational pretraining. the quantized versions of convert and multicontext convert take up only 59 mb and 73 mb, respectively, and train for 18 hours with a training cost estimate of only 85 usd. we hope that this work will motivate and guide further developments in the areas of retrieval-based task-oriented dialog and large-scale pretraining for conversational applications @xcite .