Intra-/Inter-Interaction Network with Latent Interaction Modeling for Multi-turn Response Selection (2020)

Abstract: Multi-turn response selection has been extensively studied and applied to many real-world applications in recent years. However, current methods typically model the interactions between multi-turn utterances and candidate responses with iterative approaches, which is not practical as the turns of conversations vary. Besides, some latent features, such as user intent and conversation topic, are under-discovered in existing works. In this work, we propose Intra-/Inter-Interaction Network (I ^3 ) with latent interaction modeling to comprehensively model multi-level interactions between the utterance context and the response. In specific, we first encode the intra- and inter-utterance interaction with the given response from both individual utterance and the overall utterance context. Then we develop a latent multi-view subspace clustering module to model the latent interaction between the utterance and response. Experimental results show that the proposed method substantially and consistently outperforms existing state-of-the-art methods on three multi-turn response selection benchmark datasets.

## introduction recent years have witnessed many successful real-world applications on chatbots and ai assistants, such as the xiaoice @xcite from microsoft and the e-commerce assistant alime @xcite from alibaba group, which owe to the extensive researches on dialogue systems. existing works on building conversational models mainly study generation-based @xcite or retrieval-based methods @xcite @xcite . in this work, we focus on the problem of multi-turn response selection for retrieval-based dialogue systems, which aims at selecting appropriate responses from a set of candidates as the reply for the given multi-turn utterances. measuring the matching degree between the utterance context and the candidate response is the core of multi-turn response selection task. recent works develop a variety of interaction model to enhance the utterance-response interaction from a broader @xcite or deeper perspective @xcite @xcite . empirical evidences show that iterative architectures achieve state-of-the-art performance on multi-turn response selection, such as interactionover-interaction @xcite , iterated attentive matching @xcite , and multi-hop selector @xcite . despite the effectiveness of these methods, multi-turn response selection task still remains some challenges when modeling the interaction between the utterance context and response: (i) in order to capture the interaction information between a candidate response and multi-turn utterances, most of existing iterative architectures may require deeper or more complex network structure along with the growth of the turns of conversations, which fall short to efficiently learn the multi-turn utterance representations. (ii) existing methods mainly focus on measuring the semantic relevancy between the response and the given utterance context. nevertheless, researchers observe that some latent features in the conversations, such as user intent @xcite @xcite or conversation topic @xcite @xcite , also attach great importance in dialogue systems, which have received little attention in recent multi-turn response selection studies. in this work, we propose intra-/inter-interaction network (i 3 ) with latent interaction modeling to tackle the aforementioned issues. in specific, we adopt hierarchical structure instead of iterative structure to model the multi-level interactions in the multi-turn conversation, including the intra-utterance interaction between the response and each individual utterance, and the inter-utterance interaction among the response and the overall utterance context. such comprehensive sentence representational learning enables each utterance to be encoded with rich information for mining the latent features. besides, subspace clustering @xcite @xcite , which aims to cluster the data into multiple subspaces and find a low-dimensional subspace for each class of data in an unsupervised manner, can be an effective approach to learn the latent feature representations without human-annotated labels. as for dialogue systems, the utterance context and the response can be regarded as two independent views of data @xcite , and it is required to learn the latent representation from both views in a common space to model the coherency of their latent features. inspired by latest mulit-view subspace clustering studies @xcite , we propose two kinds of latent multiview subspace clustering module, namely linear and generalized latent multi-view subspace clustering (llmsc and glmsc), to capture the latent features, which first encode the utterance and the response into view-specific latent representation respectively, and then project them to the same subspaces for multi-view clustering. finally, we aggregate the three-level interaction information, including the intra-/inter-utterance interaction and latent feature matching information, to comprehensively measure the matching degree between the utterance context and candidate response. to summarize, the main contributions of this work are as follows: (1) we propose a novel multi-turn response selection model, intra-/inter-interaction network (i 3 ), to capture the multi-level matching information by modeling the multi-turn conversations as a hierarchical structure; (2) we develop two kinds of latent multi-view subspace clustering module to model the latent feature coherency between the utterance and response; (3) experimental results show that the proposed method substantially and consistently outperforms existing state-of-the-art methods on three multi-turn dialogue benchmark datasets. ## related works existing methods for building intelligent dialogue systems can be categorized into retrieval-based methods @xcite @xcite , generation-based methods @xcite and hybrid methods @xcite . besides, current studies on conversational systems have evolved from single-turn @xcite into multi-turn scenarios @xcite . in this work, we focus on retrieval-based methods for multi-turn response selection. the key to matching the response and the given utterance context is modeling the interaction between them, which is mainly addressed by deep learning models in current studies, like cnn @xcite , rnn @xcite , and hybrid models @xcite . based on these deep neural networks, some recent works further develop diverse and effective approaches to measure the relevance between the response and the utterances, such as integrating multi-view matching information @xcite , modeling sequential utterance information @xcite , and refinement and aggregation scheme @xcite . inspired by recent progresses of transformer model @xcite , latest studies on multi-turn response selection step up to a new stage with carefully designed selfattention-based interaction networks, including deep attention matching network @xcite , multi-representation fusion network @xcite , interaction-over-interaction network @xcite , and multi-hop selector network @xcite . in this work, we facilitate the interaction modeling by considering both intra-/inter-utterance interaction with a hierarchical encoder. apart from measuring the semantic and contextual relevancy, several efforts have been made on discovering some latent features in the conversations for modeling the intent or topic coherency between the utterance and the response. yoon et al. (2018) and yoon et al. (2019) incorporate latent clustering into context-based response/answer selection models to fetch latent topic information. yang et al. (2018) utterance-1 utterance-n response self-attention layer dual-attention layer self-attention layer dual-attention layer self-attention layer intra-utterance encoder origin interaction intra-utterance interaction self-attention layer dual-attention layer inter-utterance encoder self-attention layer latent multi-view subspace clustering latent interaction aggregation inter-utterance interaction . ## conclusion in this work, we propose intra-/inter-interaction network (i 3 ) with latent interaction modeling for multiturn response selection. we propose a hierarchical transformer encoder to capture the intra-and interutterance interaction with the candidate response from both individual utterance and the overall utterance context. besides, we develop a latent multi-view subspace clustering module to model the latent feature coherency between the utterance and response. experimental results show that the proposed method substantially and consistently outperforms existing sota methods on three benchmark datasets.