Profile Consistency Identification for Open-domain Dialogue Agents (2020)

Abstract: Maintaining a consistent attribute profile is crucial for dialogue agents to naturally converse with humans. Existing studies on improving attribute consistency mainly explored how to incorporate attribute information in the responses, but few efforts have been made to identify the consistency relations between response and attribute profile. To facilitate the study of profile consistency identification, we create a large-scale human-annotated dataset with over 110K single-turn conversations and their key-value attribute profiles. Explicit relation between response and profile is manually labeled. We also propose a key-value structure information enriched BERT model to identify the profile consistency, and it gained improvements over strong baselines. Further evaluations on downstream tasks demonstrate that the profile consistency identification model is conducive for improving dialogue consistency.

## introduction despite the recent advancements in assigning attribute profiles to dialogue agents @xcite , maintaining a consistent profile is still challenging for an open-domain dialogue agent. existing works mainly emphasize the incorporation of attribute information in the generated responses @xcite @xcite . although these models have improved the response consistency by explicitly modeling the profiles, they still face the consistency issue @xcite . one important reason is that they cannot identify the consistency relations between response and profile. as shown in figure 1 , the attribute word beijing is incorporated in the first two responses, but only r 1 is semantically consistent with the speaker's profile. for example, r 2 "i also hope to visit beijing one day." implies that the speaker has never figure 1 : left: the key-value attribute profiles of the dialogue agent. right: a dialogue query with different responses that might be related to the attribute profiles. among these responses, r 1 entails the current location profile, while r 2 contradicts the profile. although r 3 does not contain the attribute word beijing, we could still understand r 3 entails the current location. in this work, we introduce a human-annotated dataset, named key-value profile identification (kvpi), with over 110k single-turn conversations and corresponding attribute profiles. three representative domains, gender, location, and constellation, are involved in the human annotation. we hire an annotation team to (1) label the relation (entailed, contradicted, or irrelevant) between each conversation and structured profile, and (2) find out the detailed attribute information in each response. with the annotated kvpi dataset, we set up different baseline models, and propose a key-value structure information enriched bert (kvbert) model, which leverages dependency structures in profiles to enrich the contextual representations. experimental results show that kvbert obtains significant improvements over strong baselines. we further test the kvbert model on two downstream tasks, including a reranking task @xcite and a consistency prediction task @xcite . evaluation results show that (1) the kvbert reranking improves response consistency, and (2) the kvbert consistency prediction has a good agreement with human annotation. our contributions are summarized as below: ## dataset preparation in this section, we describe the collection and annotation process of the kvpi dataset: (1) how we collect high-quality conversations and profiles; ## the kvpi dataset we present some examples of the final kvpi dataset in ## profile consistency identification 4.1 problem definition to equip dialogue agents with the ability to identify consistency, we need to build a profile consistency identification model. this model learns to identify the relation of {entailed, contradicted, irrele-vant} between a (profile, response) pair. formally, our goal is to learn a mapping function f, and f(p, r) ∈ {e, c, i}, where @xmath0: v 1 , ..., k n : v n }, @xmath1, w 2 , ..., w m . here p denotes the keyvalue profile, and r denotes the response with m words. e, c, i denote the consistency relations. ## experiments in this section, we first evaluate the performance of the proposed kvbert model on identifying profile consistency. after that, we test the trained kvbert model on two downstream tasks, including a reranking task and a consistency prediction task, to analyze how well the proposed approach performs under practical applications. ## related work this work is closely related to the researches in natural language inference @xcite . nli aims to determine whether a natural language hypothesis can be inferred from a natural language premise @xcite @xcite . besides the natural language evidence, @xcite and @xcite proposed to use images as the evidence for statement verification under the multi-modal setting. a more recent related work is the @xcite , who proposed to use semi-structured wikipedia tables as evidence. the difference between our work and @xcite is noticeable: open-domain dialogues have unique language patterns, and the key-value profiles are highly structured, as analyzed in sec 4.2. to the best of our knowledge, this is the first work that explores the identification of consistency between dialogue responses and structured profiles. another line of research related to this work is the personalized dialogue generation task @xcite @xcite . this task seeks to improve personality consistency by incorporating persona information in the generated responses. for this purpose, several personalized dialogue datasets have been introduced in recent years, such as per-sonachat @xcite and personaldialog @xcite . these datasets successfully inform models of how to incorporate attribute related information in the responses, but still can not teach models how to identify the consistency relations between their response and profile. ## conclusion and discussion in this work, we introduce a large-scale annotated dataset to facilitate the study of profile consistency identification in open-domain dialogues. we leverage the structure information in profiles to enrich the bert representations and obtain significant performance improvements over strong baselines. we further test the proposed method on two downstream tasks. evaluation results show the effectiveness of the proposed approach. we believe kvpi will be a useful resource for the research of open-domain dialogue consistency. although there has been a lot of dialogue generation models in this field, most of them still can't understand the consistency relationship in the generation process. one of the major bottlenecks is the lack of data. because the kvpi dataset has paired key-value profiles and dialogues, it can also be a high-quality resource for personalized dialogue generation tasks. furthermore, because we have fine-grained consistency labels, this dataset also provides an opportunity to leverage natural language understanding models to assist dialogue generation models. we hope that the data will aid training dialogue agents to be more consistent. where to find a good man? reranking 可是男人在哪里？ but where is the man? 你还未娶，我不着急嫁 you haven't married a girl. i can wait. 我要找女的嫁 i'd rather marry a girl