Response Selection for Multi-Party Conversations with Dynamic Topic Tracking (2020)

Abstract: While participants in a multi-party multi-turn conversation simultaneously engage in multiple conversation topics, existing response selection methods are developed mainly focusing on a two-party single-conversation scenario. Hence, the prolongation and transition of conversation topics are ignored by current methods. In this work, we frame response selection as a dynamic topic tracking task to match the topic between the response and relevant conversation context. With this new formulation, we propose a novel multi-task learning framework that supports efficient encoding through large pretrained models with only two utterances at once to perform dynamic topic disentanglement and response selection. We also propose Topic-BERT an essential pretraining step to embed topic information into BERT with self-supervised learning. Experimental results on the DSTC-8 Ubuntu IRC dataset show state-of-the-art results in response selection and topic disentanglement tasks outperforming existing methods by a good margin.

## introduction in recent years, with the influx of deep learning methods in natural language processing (nlp), there has been a lot of interests in building effective task-oriented dialogue systems that can assist people in real-world business such as booking tickets, ordering food and solving technical issues @xcite . retrieval-based response generation that selects a suitable response from a pool of candidates (pre-existing human responses) has become a popular approach to framing dialog. compared to the generation-based systems that generate novel utterances @xcite , retrieval-based systems produce fluent, grammatical and informative responses (weston et al., 2018; henderson et al., 1 code will be available at https://github.com/ salesforce/topicbert . figure 1 : a (truncated) multi-party conversation from ubuntu irc log. curved arrows show the 'reply-to' links between utterances. we use different colors to represent different conversation topic clusters. 2019). also compared to the traditional modular approach, it does not rely on dedicated modules for language understanding, dialog management, and generation, thus simplifying the system design. due to these reasons, retrieval-based systems have been widely adopted in commercial dialogue systems @xcite . initially, researchers considered response selection in single-turn conversations, where only the last input utterance is considered as the context query @xcite . more recent work deals with multi-turn context, which shows improvements over the single-turn context @xcite @xcite @xcite . these methods typically aim to encode the context and the candidate responses in a joint semantic space by capturing short and long range dependencies, and then retrieve the most relevant response by matching the query representation against each candidate's representation through attentions. however, most of these works are limited to only two-party conversations. as dialogue research progresses, it is necessary to study the more generic multi-party multi-turn scenario, which has become very common (e.g., slack, whatsapp) with the advent of internet and mobile devices, and posits a unique set of challenges for the dialog models @xcite . multiple ongoing conversations seem to occur more naturally in multi-party conversations. for example, consider the conversation excerpt in figure 1 among three participants, taken from the ubuntu irc corpus. there are three ongoing conversation topics as highlighted by different color, and the participants contribute to multiple topics simultaneously (e.g., nafallo participates in three and danhunt participates in two). an effective response selection method should model such complex conversational topic dynamics in the context, for which existing methods are deficient. in particular, a proper response should match with its context in terms of the same conversation topic, while ignoring other non-relevant topics. to address the aforementioned challenges in multi-party multi-turn dialog, we frame response selection as a dynamic topic tracking task with the intuition that the topic should remain the same as we go from the context to the response. our formulation is also supported by the segmented discourse representation theory (sdrt) of conversations @xcite . based on this new formulation, we propose a novel architecture that can incorporate other related dialog tasks such as conversation disentanglement, enabling multi-task learning in a unified framework. crucially, our formulation of the task needs to encode only two utterances at a time, thus allowing efficient encoding via large pretrained models like bert @xcite . furthermore, it facilitates pretraining of bert-like models on topic related sentence pairs to incorporate topic relevance in pretraining, which can be done on large dialog corpora with self-supervised objectives, requiring no manual topic annotations, and can benefit not only response selection but also other dialog tasks. in summary, our contributions are: based on the similarity of these three tasks, the objective is to match topic (topic prediction) between context utterance and response and track response's topic (topic disentanglement) across contexts to select an appropriate response. ## task formulation our topic-bert framework combines response selection task with two auxiliary tasks, which are topic prediction and topic disentanglement. ## our topic-bert framework our framework for response selection aims to track how the conversation topics change from one utterance to another and use it for ranking the candidate responses. as shown in fig. 2 , we encode an utterance u k from the context c @xmath0, u 2 , . . . , u n } along with a candidate response r i,j using our pretrained topic-bert encoder ( §4.1). the contextual token representations in topic-bert encode topic relevance between the tokens of u k and the tokens of r i,j , while the [cls] representation captures utterance-level topic relevance. we use the [cls] representation as query to attend over the token representations to further enforce topic relevance in the attended topic vector tk . we repeat this encoding process for the n utterances in the context c i to get n different topic vectors t @xmath1, . . . , t n } that model r i,j 's topic relevance to the each of context utterances. these topic representations are then used for the prediction tasks -topic prediction, disentanglement, and response selection. response selection is our main task, while the other two tasks are auxiliary and optional. since our topic-bert encodes two utterances at a time, the encoding process is efficient and can be used to encode larger context. the core component of our framework is the topic-bert pretraining as we describe next. ## experiments in this section, we present our experiments, including the datasets, experimental setup, evaluation metrics, and the results with analysis.