The World is Not Binary: Learning to Rank with Grayscale Data for Dialogue Response Selection (2020)

Abstract: Response selection plays a vital role in building retrieval-based conversation systems. Despite that response selection is naturally a learning-to-rank problem, most prior works take a point-wise view and train binary classifiers for this task: each response candidate is labeled either relevant (one) or irrelevant (zero). On the one hand, this formalization can be sub-optimal due to its ignorance of the diversity of response quality. On the other hand, annotating grayscale data for learning-to-rank can be prohibitively expensive and challenging. In this work, we show that grayscale data can be automatically constructed without human effort. Our method employs off-the-shelf response retrieval models and response generation models as automatic grayscale data generators. With the constructed grayscale data, we propose multi-level ranking objectives for training, which can (1) teach a matching model to capture more fine-grained context-response relevance difference and (2) reduce the train-test discrepancy in terms of distractor strength. Our method is simple, effective, and universal. Experiments on three benchmark datasets and four state-of-the-art matching models show that the proposed approach brings significant and consistent performance improvements.

## introduction building intelligent conversation systems @xcite is gaining more and more attention in recent years. a core module in such kind of conversation systems is response selection @xcite @xcite : identifying the best response from a set of possible candidates given a dialogue context, i.e., conversation history. for dialogue context between speakers a and b relevance a: would you please share some useful experience for improving spoken english? b: sure! watching english movies helped a lot. a: agreed. i watched friends many times. b: me too! i bought the dvds and they went broken due to my frequent use. ground truth g: hah! then your english should be very good! + distractor response during training r1: why didn't the british police come? the response selection problem, the trendy practice is to build neural matching models @xcite @xcite @xcite for scoring the adequacy of individual response candidates in the dialogue context. most prior works on this topic focus on fine-grained text encoding and better interactions between dialogue context and response candidates, typically via sophisticated and powerful matching networks @xcite @xcite . despite their differences, in almost all these previous works, the matching models are trained with binary classification objective. each response in the training data is either labeled positive (i.e., a correct response to the dialogue context) or negative (i.e., an incorrect response). often, the negative responses are automatically constructed by random sampling. one limitation of the above training strategy is that this formalization downplays the nuance of fine-grained response quality; the matching model is only informed to predict a binary label, either correct or incorrect. however, the quality of possible response candidates may be quite diverse, thus letting the matching model be aware of which response candidates are more incorrect or less incorrect than others may more effectively increase the model capacity. another limitation is that in real-world scenarios the matching models are often confronted with more difficult tasks: to select the best response from a set of strong response candidates instead of random ones. an example is given in one natural solution to the above problems is to collect grayscale data for training; if we consider the quality of all possible response candidates falls in the interval [0, 1], the golden-truth and random responses usually cover the two endpoints only, and our goal is to obtain a list of grayscale responses locate in between 0 and 1. however, grayscale data are hard to obtain in reality owing to the expense of human annotation and the subjectivity of individual human annotators. in this work, we propose to automatically construct grayscale data from standard dialogue datasets, where only golden dialogue context and response pairs are provided. to meet this goal, we resort to off-the-shelf retrieval algorithms and generation models. our idea is inspired by the observation that, in most cases, the responses from retrieval models or generation models are better than randomly sampled ones but worse than the ground-truth response. we believe that this progressive relationship, such as "ground @xmath0, can be utilized for training a better matching model. concretely, we propose a multi-level ranking objective to make full use of such relationships. our multi-level ranking objective jointly combines multiple binary contrastive estimations. in addition, the grayscale data partly simulates the real-world response distractors and thus reduces the gap between training and testing, leading to a better distinguishing ability for strong response distractors. our method is simple, effective, and orthogonal to prior efforts for modeling designs. it can be conveniently implemented with most existing matching models. experimental results on four state-of-the-art matching models and three benchmark datasets demonstrate that our new training approach leads to remarkable performance improvement consistently. ## background early research for response selection is devoted to single-turn conversations @xcite @xcite . recently, researchers have started to study on multi-turn conversations @xcite @xcite . in the current literature, the task of response selection is formulated as follows. given a dialogue dataset @xmath1, r i )}, where c i represents a dialogue context, and r i is the human-written ground-truth response. the goal is to build a matching model s(•, •) from d so that s(c, r) accurately measures the adequacy of a response candidate r for a dialogue context c. rapid progress has been made for building such matching models in recent years. concretely, various neural architectures @xcite @xcite @xcite have been proposed for fine-grained text encoding and better dialogue context and response interactions modeling. to train such matching models, binary-labeled training sets are constructed @xcite @xcite : the humanwritten ground-truth response is designated as positive instances (labeled as 1), and a set of randomly sampled responses n i are treated as negative ones (labeled as 0). the learning objective of s(•, •) is then to maximize the following binary classification loss function: different from previous works, our study questions the effectiveness of the binary-labeled training data and the corresponding binary classification objective. we argued that the binary classification paradigm is sub-optimal as most of the randomly sampled negative responses are distant from the corresponding positive responses in terms of matching degree, which could lead to serious drawbacks when some strong distractors are presented during testing @xcite . our work starts with enriching the range of the negative sample set n i in terms of response quality and leads to a simple but new learning strategy that aimed at capturing more fine-grained response quality differences. ## related work some researchers also studied how to improve the performance of existing matching models with a better learning method. @xcite proposed to leverage a seq2seq model as a weak annotator to assign a score for each response candidate of the dialogue and learn matching models through the scores. @xcite introduced the coteaching framework @xcite for eliminat-ing the effect of training noises. the learning approach maintains two matching models and makes them teach each other. @xcite attempted to neglect the effect of false negatives and trivial true responses by adopting four negative sampling strategies to choose negative samples during training dynamically. different from those previous works, our approach makes use of grayscale data from heterogeneous sources and learns progressive quality relationships. in addition, our work enhances retrieval models with generation models, which is on par with recent attempts @xcite to strengthen generation models via retrieval models.