K d C onv: A C hinese Multi-domain Dialogue Dataset Towards Multi-turn Knowledge-driven Conversation (2020)

Abstract: The research of knowledge-driven conversational systems is largely limited due to the lack of dialog data which consists of multi-turn conversations on multiple topics and with knowledge annotations. In this paper, we propose a Chinese multi-domain knowledge-driven conversation dataset, KdConv, which grounds the topics in multi-turn conversations to knowledge graphs. Our corpus contains 4.5K conversations from three domains (film, music, and travel), and 86K utterances with an average turn number of 19.0. These conversations contain in-depth discussions on related topics and natural transition between multiple topics. To facilitate the following research on this corpus, we provide several benchmark models. Comparative results show that the models can be enhanced by introducing background knowledge, yet there is still a large space for leveraging knowledge to model multi-turn conversations for further research. Results also show that there are obvious performance differences between different domains, indicating that it is worth further explore transfer learning and domain adaptation. The corpus and benchmark models are publicly available.

## introduction it has been a long-term goal of artificial intelligence to deliver human-like conversations, where background knowledge plays a crucial role in the success of conversational systems @xcite @xcite . in taskoriented dialog systems, background knowledge is defined as slot-value pairs, which provides key information for question answering or recommendation, and has been well defined and thoroughly studied @xcite . in open-domain conversational systems, it is important but challenging to leverage background knowledge, which is represented as either knowledge graphs @xcite or unstructured texts @xcite , for making effective interactions. recently, a variety of knowledge-grounded conversation corpora have been proposed @xcite @xcite @xcite @xcite to fill the gap where previous datasets do not provide knowledge grounding of the conversations @xcite @xcite . cmu dog @xcite , india dog @xcite , and wizard of wikipedia @xcite demonstrate attempts for generating informative responses with topic-related wikipedia articles. however, these datasets are not suitable for modeling topic transition or knowledge planning through multi-turn dialogs based on the relations of topics. opendialkg @xcite and duconv @xcite use knowledge graphs as knowledge resources. nevertheless, the number of topics is limited to one @xcite or two @xcite , which is not sufficient for diversified topic transition in humanlike conversations. therefore, these knowledgegrounded dialog datasets still have limitations in modeling knowledge interactions foot_0 in multi-turn conversations. in this paper, we propose kdconv, a chinese multi-domain dataset towards multi-turn knowledge-driven conversation, which is suitable for modeling knowledge interactions in multi-turn human-like dialogues, including knowledge planning, knowledge grounding, knowledge adaptations, etc. kdconv contains 86k utterances and conversation (music) knowledge triple head entity relation tail entity user1: 知道《飞得更高》这首歌吗？ do you know the song 'flying higher'? user2: 知道呀，这首歌入选了中歌榜中国年度最受华人欢迎十大金曲。 yes, this song has been selected in the top ten most popular songs in china. flying higher information … selected in the top ten most popular songs in china… … user1: 具体的发行时间你记得吗？ do you remember the exact release date? user2: 记得，是在 2005 年 3 月 19 日。 yes. it is march 19, 2005. flying higher release date march 19, 2005 user1: 我觉得这首歌算是汪峰的经典之曲。 i think it is one of the classic songs of wang feng. original singer wang feng user2:我也那么认为，编曲填词都由他自己完成，真的算是经典之作了。 so do i. the arrangement and lyrics of the music are all completed by himself. it's really a classic. arrangment lyrics user1: 说到他真的很了不起，在音乐方面获得很多大奖，我能说上来的 就有第 12 届音乐风云榜年度最佳男歌手奖。 he is really amazing and has won many awards in music, such as the 12th music awards of the year award for best male singer. wang feng main achievements the 12th music awards of the year award for best male singer … user1: 那他的歌曲除飞得更高，你还喜欢哪首？ so which song do you like besides 'flying higher'? user2: 再喜欢的就是《怒放的生命》这首歌了，听的感觉特别好，减压。 i like 'blooming life'. i feel great and decompression. wang feng representative works fireworks, brave heart, flying higher, blooming life… user1: 啊，这首歌我也很喜欢，也都是由他自己作词作曲并演唱。 oh, i like this song, too. he wrote and sang it by himself. blooming life information 'blooming life' is a song sung, written and composed by wang feng… the song won the best song of the year award in the 13th global chinese music list. user2: 是的， 该曲也获得了 13 届全球华语音乐榜中榜年度最佳歌曲奖。 yes, and the song also won the best song of the year award in the 13th global chinese music list. knowledge graph … selected in the top ten most popular songs in china in 2015… flying higher 'blooming life' is a song sung, written and composed by wang feng… the song won the best song of the year award in the 13th global chinese music list. wang feng blooming life march 19, 2005 the 12th music awards of the year award for best male singer original singer arrangment, lyrics representative work information information main achievements release date the underlined text is the related knowledge that is utilized in conversation. the italic text and circles are topics (refer to the distinct head entities in the knowledge triples and the central nodes with degree greater than 1 in the knowledge graph) in this dialogue. in summary, this paper makes the following contributions: ## related work recently, open-domain conversation generation has been largely advanced due to the increase of publicly available dialogue data @xcite @xcite . however, the lack of annotation of background information or related knowledge results in significantly degenerated conversations, where the text is bland and strangely repetitive @xcite . these models produce conversations that are substantially different from those humans make, which largely rely on background knowledge. to facilitate the development of conversational models that mimic human conversations, there have been several knowledge-grounded corpora proposed. some datasets @xcite @xcite @xcite collect dialogues and label the knowledge annotations using ner, string match, artificial scoring, and filtering rules 3 https://github.com/thu-coai/kdconv based on external knowledge resources @xcite . however, mismatches between dialogues and knowledge resources introduce noises to these datasets. to obtain the high-quality knowledgegrounded datasets, some studies construct dialogues from scratch with human annotators, based on the unstructured text or structured knowledge graphs. for instance, several datasets @xcite @xcite have human conversations where one or both participants have access to the unstructured text of related background knowledge, while opendi-alkg @xcite and duconv @xcite build up their corpora based on structured knowledge graphs. in cmu dog @xcite utilizes 30 wikipedia articles about popular movies as grounded documents, which explores two scenarios: only one participant has access to the document, or both have. also using wikipedia articles, however, wizard of wikipedia (wow) @xcite covers much more dialogue topics (up to 1,365), which puts forward a high demand for the generalization ability of dialog generation models. one other difference from cmu dog is that in wow, only one participant has access to an information retrieval system that shows the worker paragraphs from wikipedia possibly relevant to the conversation, which is unobservable to the other. in addition to the unstructured text, india dog @xcite uses fact tables as background resources. the idea of using structured knowledge to construct dialogue data is also adopted in opendialkg @xcite , which has a similar setting to kdconv. opendialkg contains chit-chat conversations between two agents engaging in a dialog about a given topic. it uses the freebase knowledge base @xcite as background knowledge. in opendialkg, the entities and relations that are mentioned in the dialog are annotated, and it also covers multiple domains (film, books, sports, and music). however, the limitation is that there are much fewer turns in a conversation, and the whole dialogue is restricted to only one given topic, which is not suitable for modeling topic transition in human-like conversations. to the best of our knowledge, duconv @xcite is the only existing chinese human-labeled knowledge-grounded dialogue dataset. duconv also utilizes unstructured text like short comments and synopsis, and structured knowledge graphs as knowledge resources. given the knowledge graph, it samples two linked entities, one as the transitional topic and the other as the goal topic, to construct a conversation path. this path is used to guide participants toward the goal of the dialogue, which, as argued in @xcite , can guide a model to deliver proactive conversations. however, the existence of the target path is inconsistent with an open dialogue in reality because humans usually do not make any assumption about the final topic of a conversation. beyond that, the knowledge graph and the goal knowledge path are only annotated for the whole dialogue, which cannot provide explicit supervision on knowledge interactions for conversational models. ## dataset collection kdconv is designed to collect open-domain multiturn conversations for modeling knowledge interactions in human-like dialogues, including knowledge planning, knowledge grounding, knowledge adaptations, etc. however, the open-domain background or commonsense knowledge is too large in scale (e.g., there are over 8 million concepts and 21 million relations in conceptnet @xcite ). thus, it is costly and time-consuming to collect multi-turn conversations from scratch based on such large-scale knowledge. kdconv is proposed as one small step to achieve this goal, where we narrowed down the scale of background knowledge to several domains (film, music, and travel) and collected conversations based on the domainspecific knowledge. kdconv contains similar domains (film and music) and dissimilar domains (film and travel) so that it offers the possibility to investigate the generalization and transferability of knowledge-driven conversational models with transfer learning or meta learning @xcite . in the following subsections, we will describe the two steps in data collection: (1) constructing the domain-specific knowledge graph; (2) collecting conversation utterances and knowledge interactions by crowdsourcing. ## conclusion and future work in this paper, we propose a chinese multi-domain corpus for knowledge-driven conversation generation, kdconv. it contains 86k utterances and 4.5k dialogues, with an average number of 19.0 turns. each dialogue contains various topics and sentence-level annotations that map each utterance with the related knowledge triples. the dataset provides a benchmark to evaluate the ability to model knowledge-driven conversations. in addition, kd-conv covers three domains, including film, music, and travel, that can be used to explore domain adaptation or transfer learning for further research. we provide generation-and retrieval-based benchmark models to facilitate further research. extensive experiments demonstrate that these models can be enhanced by introducing knowledge, whereas there is still much room in knowledge-grounded conversation modeling for future work.