How to Represent Context Better? An Empirical Study on Context Modeling for Multi-turn Response Selection (2022)

Abstract: Building retrieval-based dialogue models that can predict appropriate responses based on the understanding of multi-turn context messages is a challenging problem. Early models usually concatenate all utterances or independently encode each dialogue turn, which may lead to an inadequate understanding of dialogue status. Although a few researchers have noticed the importance of context modeling in multi-turn response prediction, there is no systematic comparison to analyze how to model context effectively and no framework to unify those methods. In this paper, instead of configuring new architectures, we investigate how to improve existing models with a better context modeling method. Specifically, we heuristically summarize three categories of turn-aware context modeling strategies which model the context messages from the perspective of sequential relationship, local relationship, and query-aware manner respectively. A Turn-Aware Context Modeling (TACM) layer is explored to flexibly adapt and unify these context modeling strategies to several advanced response selection models. Evaluation results on three public data sets indicate that employing each individual context modeling strategy or multiple strategies can consistently improve the performance of existing models.

## introduction recently, building a dialogue system for open domain human-machine conversation is attracting more and more attention due to both availability of large-scale human conversation data and powerful models learned with neural networks. existing work on building a conversational system includes generation-based methods and retrievalbased methods. a generation-based model directly * corresponding author: dongyan zhao. real-world dialogues usually comprise multiple turns, where a retrieval model should select the most proper response by measuring the matching degree between multi-turn dialogue context and a number of response candidates. the key problem is how to make better use of multi-turn context information. currently, there emerge two lines of research to represent the multi-turn dialogue context. one is to model each turn of utterance individually first and then aggregate a sequence of utteranceresponse matching features to get a final score @xcite @xcite @xcite , which are known as the representation-matching-aggregation paradigm. the other line is to concatenate all turns of utterances into a long sequence first and make them fully interact with each other by rnns @xcite @xcite or transformer layers @xcite @xcite . in particular, recently models based on pre-trained language models (plms) such as bert or sa-bert @xcite conduct multi-turn context modeling and response matching in a unified process. these mainstream methods, including fully concatenating all utterances or independently encoding each dialogue turn, equally represent the information of each dialogue element and ignore the characteristics of multi-turn dialogue context, which may lead to sub-optimal context representations and response matching features. recently, researchers have begun to notice the importance of explicitly modeling the multi-turn dialogue context based on the characteristics of multi-turn dialogue context, including using of natural sequential relationship between dialogue turns @xcite or using the last turn of dialogue context to guide the modeling process of previous turns @xcite . however, there is no systematic comparison to analyze how to effectively model the multi-turn dialogue context considering characteristics of dialogue model and no framework to unify those methods for retrieval-based dialogues. in this paper, instead of configuring new architectures, we investigate how to improve the performance of existing matching models with better context modeling methods. following this idea, we heuristically summarize three categories of turn-aware context modeling strategies which model context messages from the perspective of sequential relationship, local relationship, and queryaware manner respectively. to compare those methods, we apply them on several representative response selection models through a turn-aware context modeling (tacm) layer, which allows different context modeling strategies to be flexibly applied to dialogue models. to verify the effectiveness of the framework, we choose three representative multi-turn response selection models as our matching models, and conduct experiments on three public data sets including ubuntu dialogue corpus @xcite , douban conversation corpus @xcite , and e-commerce dialogue corpus @xcite . based on a series of experiments, we find query-aware context modeling is the best strategy and employing multiple context modeling strategies can consistently improve the performance of response selection. besides, we also observe that our tacm layer can improve the capability of modeling long context. we hope our empirical comparison can shed light on future research on this line of work. our contributions in this paper are four-fold: ## related works retrieval-based models design a discriminative model to measure the matching degree between a human input and a response candidate for response selection. early studies mainly focus on single-turn context-response matching @xcite @xcite . recently, researcher have devoted themselves to the multi-turn scenario. several methods concatenate all turns of utterances into a long sequence first and then make them fully interact with each other by rnns @xcite @xcite or transformer layers @xcite . in addition to these methods, some researchers construct dialogue models with a representation-matchingaggregation paradigm. such approaches encode each turn of utterance individually first and then aggregate a sequence of utterance-response matching features to get a final score. representative methods including sequential matching network (smn) @xcite , deep attention matching network (dam) @xcite and multi-hop selector network (msn) @xcite . as an important problem in dialogue systems, multi-turn context modeling has raised great interests in recent years. especially for generationbased methods, various models adopt hierarchical encoder-decoder framework to model all context sentences @xcite @xcite . @xcite compare various methods to get a global representation for the context. @xcite propose recosa where attention weights between each context and response representations are computed and used in further decoding process. the problem is less explored in existing retrieval-based methods. @xcite concatenate the last utterance to other turns, and then use gated self attention to obtain utterance representation. @xcite use multi-hop selectors to select useful information in dialogue history. these methods only consider modeling one type of characteristic of multi-turn dialogue context. besides, there is no systematic comparison to analyze how to model context effectively and no framework to unify those methods. therefore, we consider exploring how to improve the existing models with a better context modeling method in this paper. specifically, we summarize three categories of turn-aware context modeling strategies and conduct an empirical study on context modeling for multi-turn response selection. ## conclusion this paper investigates how to improve the performance of existing matching models with a better context modeling method. empirical results on three benchmarks indicate that query-aware context modeling is the best strategy and employing multiple context modeling strategies can consistently improve the performance of existing response selection models. additionally, our tacm layer can improve the capability of modeling long context.