Fine-grained Post-training for Improving Retrieval-based Dialogue Systems (2021)

Abstract: Retrieval-based dialogue systems display an outstanding performance when pre-trained language models are used, which includes bidirectional encoder representations from transformers (BERT). During the multi-turn response selection, BERT focuses on training the relationship between the context with multiple utterances and the response. However, this method of training is insufficient when considering the relations between each utterance in the context. This leads to a problem of not completely understanding the context flow that is required to select a response. To address this issue, we propose a new fine-grained post-training method that reflects the characteristics of the multi-turn dialogue. Specifically, the model learns the utterance level interactions by training every short context-response pair in a dialogue session. Furthermore, by using a new training objective, the utterance relevance classification, the model understands the semantic relevance and coherence between the dialogue utterances. Experimental results show that our model achieves new state-of-the-art with significant margins on three benchmark datasets. This suggests that the fine-grained post-training method is highly effective for the response selection task.

## introduction constructing a dialogue system that can naturally and consistently interact with humans is currently a popular research topic. there are two approaches for the implementation of a dialogue system: generation-based and retrieval-based methods. the latter approach aims to select the correct response among the response candidates. in the initial multi-turn response selection, @xcite proposed leveraging rnn to match the dialogue context with a response. later, with the advent of the attention mechanism @xcite @xcite , multiturn response selection models that use the attention mechanism have been proposed @xcite . recently, pre-trained language models, such as bidirectional encoder representations from transformers, bert, have been applied to a variety of response selection models @xcite @xcite , and they have shown excellent performance. recently, pre-trained language models have been widely used in several natural language processing areas, such as question answering and dialogue systems. one of the best pre-trained language models, bert @xcite , is initially pre-trained on a large and general domain corpus, and then it is fine-tuned to adapt to specific tasks. since bert is pre-trained with general data, its performance can be improved by post-training to adapt to domain-specific data. some previous studies @xcite proposed a posttraining method that can learn domain data before fine-tuning for a task. in the previous studies, the models were post-trained using domain-specific task data with the same pre-training objectives as bert, masked language model (mlm) and next sentence prediction (nsp). to develop a new post-training method that is suitable for dialogue, we propose a simple but powerful fine-grained post-training method. the new post-training method has two learning strategies. the first is to train the model by dividing the entire dialogue into multiple short context-response pairs. the second is to train the model with a new objective called utterance relevance classification, which classifies the relation between given utterances and the target utterance into more fine-grained labels. the dialogue consists of a context that includes multiple utterances and a response with one utterance. there are two advantages to learning the dialogue by dividing it into multiple new short context-response pairs, rather than learning with the entire context-response pair during post-training. first, the model can learn the interaction between internal utterances, which is overlooked in the previous training methods. the previous multi-turn response selection models focus on identifying the associated information between a context with multiple utterances and the response. to understand the associated information, bert takes the whole context as input to represent the relationship between the context and the response, instead of gradually expanding and learning the relationship between the utterances inside the context. the relationship between the entire context and response can be learned through self-attention. however, the relationship between the utterances in the dialogue is easily overlooked. to address this issue, we divide the entire dialogue into multiple short context-response pairs. since each pair consists of internal utterances, the model can learn the utterancelevel interactions. the second advantage is that the model can capture the relationship between the utterances more accurately. in general, the utterances that are related to the response are located close to the response. as short context-response pairs consist only of utterances that are close to the response, more fine-grained training is possible. another strategy of fine-grained post-training is that it involves using a new training objective that is called the utterance relevance classification (urc). in the case of the nsp used in bert, the model distinguishes whether the target utterance is random or the next. as mentioned by @xcite , the model trained with the nsp can easily learn the topic prediction that distinguishes the semantic meaning of the utterances. however, it lacks coherence prediction that distinguishes whether the selected utterance is consecutive. in the case of sentence ordering prediction (sop) used in @xcite , the coherence between the utterances is well learned because the order of the two sequences is trained. however, topic prediction is relatively insufficient because the two sequences are semantically similar. as it is important to distinguish between semantically similar utterances in the multi-turn dialogue and determine whether the selected utterances are consecutive, we propose urc, which classifies the target utterance into three categories (random, semantically similar, next) to learn the topics and coherence. the contributions of our study are summarized as follows: the existing methods for building dialogue systems can be categorized into two groups: those with a retrieval-based approach @xcite @xcite and those with a generation-based approach @xcite @xcite . recent studies have focused on the multi-turn retrieval dialogue system where the system selects the most appropriate response when a multi-turn dialogue context is provided. lowe et al. (2015) proposed a new benchmark dataset called the ubuntu internet relay chat (irc) corpus v1 and a rnn-based baseline model. kadlec et al. (2015) suggested a dual encoder-based model that attempts to effectively encode the context and response by using lstm and cnn as encoder. with the advent of the attention mechanism @xcite @xcite , models such as the deep attention matching network @xcite , which applied the attention mechanism to the response selection dialogue system, have been proposed. @xcite adapted the natural language inference model to the response selection task. @xcite performed a deep interaction between the context and the response through multiple interaction blocks. @xcite improved the performance by controlling the dialogue context information with a multi-hop selector. the pre-trained language models have shown an impressive performance in the response selection @xcite @xcite . one of those, bert, is a bidirectional transformer-based encoder that has multiple layers. we use the publicly opened bert base model in which the number of layers, attention head, and size of the hidden state are 12, 12, and 768, respectively. there are a variety of training objectives for the pre-trained language models. bert uses two training objectives: mlm and nsp. the former randomly masks 15% of the tokens that are predicted by the model. this method of training aims for the model to learn the overall contextual representation of a given text. in the latter method, the model is given two sequences of text: a and b. the model is trained to determine if sequence b is the next sequence after sequence a. the model takes the input, sequences a and b, separated by the special token sep. the model uses the segment embeddings of 0 for sequence a and 1 for sequence b. then, by using the cls token, the model predicts the relationship between sequences a and b. al-bert @xcite uses sentence ordering prediction (sop) instead of nsp as the training objectives. the sop distinguishes whether the order of sequences a and b is correct or if they have been swapped. the post-training method, which helps the model understand a certain domain, was introduced in the response selection task @xcite @xcite @xcite . in addition to domain adaptation, the post-training method has the advantage of data augmentation because it learns the relationship between the two sequences in the dialogue session with the nsp. however, the method does not reflect the conversational characteristics because it merely follows bert's pre-training method. to address this issue, we propose a novel post-training method that is suitable for a multi-turn dialogue. the proposed method achieved better performance in comparison to the previous post-training. ## conclusion in this paper, we have proposed a new fine-grained post-training method that is suitable for the multiturn dialogue. the proposed method allows the matching model to learn the semantic relevance and the coherence of the utterances in the dialogue, and it improves the model's capability to select the appropriate response. the experimental results on the three benchmark datasets demonstrate our post-training method's superiority for the response selection. from this, our model achieved a new state-of-the-art performance for all three benchmarks. in the future, we plan to research new posttraining methods that are suitable for a variety of tasks, such as question answering and dialogue generation.