Is a Knowledge-based Response Engaging?: An Analysis on Knowledge-Grounded Dialogue with Information Source Annotation (2023)

Abstract: Currently, most knowledge-grounded dialogue response generation models focus on reflecting given external knowledge. However, even when conveying external knowledge, humans integrate their own knowledge, experiences, and opinions with external knowledge to make their utterances engaging. In this study, we analyze such human behavior by annotating the utterances in an existing knowledge-grounded dialogue corpus. Each entity in the corpus is annotated with its information source, either derived from external knowledge (database-derived) or the speaker’s own knowledge, experiences, and opinions (speaker-derived). Our analysis shows that the presence of speaker-derived information in the utterance improves dialogue engagingness. We also confirm that responses generated by an existing model, which is trained to reflect the given knowledge, cannot include speaker-derived information in responses as often as humans do.

## introduction more and more dialogue research has utilized external knowledge to enable dialogue systems to generate rich and informative responses @xcite @xcite @xcite . the major focus of such research is in how to select appropriate external knowledge and reflect it accurately in the response @xcite @xcite . however, as shown in figure 1 1 , a good speaker not only informs the dialogue partner of external knowledge but also incorporates his or her own knowledge, experiences, and opinions effectively, which makes the dialogue more engaging. the extent to which models specializing in reflecting 1 examples of dialogues presented in this paper are originally in japanese and were translated by the authors. the first of the worldwide hit movies about the pirates' struggle review figure 1 : an example of japanese movie recommendation dialogue @xcite . the table above the recommender's utterance indicates the external knowledge used in that utterance. the recommender incorporates not only database-derived information but also speaker-derived information. in this study, we first analyze how humans incorporate speaker-derived information by annotating the utterances in an existing knowledge-grounded dialogue corpus. each entity in the utterances is annotated with its information source, either derived from external knowledge (database-derived) or the speaker's own knowledge, experiences, and opinions (speaker-derived). the analysis of the annotated dataset showed that engaging utterances contained more speaker-derived information. in addition, we train a bart-based response generation model in a standard way, i.e., by minimizing perplexity, and investigate the extent to which it incorporates speaker-derived information. the result showed that the response generation model did not incorporate speaker-derived information into their utterances as often as humans do. this result implies that minimizing perplexity is insufficient to increase engagingness in knowledgegrounded response generation and suggests room for improvement in the training framework. ## information source annotation this section describes the annotation scheme for information sources and the annotation results. ## analysis of human utterances we analyze human utterances at the dialogue level and utterance level. ## analysis of system utterances we investigate the distribution of information source labels in the responses of the model trained on the knowledge-grounded dialogue dataset. first, we train a response generator ( §4.1) with the dialogue contexts and external knowledge as input and responses as output. next, an information source classifier ( §4.2) is trained with responses and external knowledge as input and information source labels as output. then, the information source classifier infers the information source labels for the system responses generated by the response generator. finally, we analyze the distribution of inferred information source labels. ## conclusion we analyzed the distribution of speaker-derived information in human and system responses in the knowledge-grounded dialogue. the analysis showed that the use of speaker-derived information, as well as external knowledge, made responses more engaging. we also confirmed that the response generation model trained in a standard way generated less speaker-derived information than humans. it is difficult to make good use of speaker-derived information by simply minimizing the perplexity of the model because a wide variety of speakerderived information appears in each dialogue. we hope our published annotated corpus becomes a good launch pad for tackling this issue.