Focus on F o C us: Is F o C us focused on Context, Knowledge and Persona? (2022)

Abstract: Rather than continuing the conversation based on personalized or implicit information, the existing conversation system generates dialogue by focusing only on the superficial content. To solve this problem, FoCus was recently released. FoCus is a persona-knowledge grounded dialogue generation dataset that leverages Wikipedia’s knowledge and personal persona, focusing on the landmarks provided by Google, enabling user-centered conversation. However, a closer empirical study is needed since research in the field is still in its early stages. Therefore, we fling two research questions about FoCus. “Is the FoCus whether for conversation or question answering?” to identify the structural problems of the dataset. “Does the FoCus model do real knowledge blending?” to closely demonstrate that the model acquires actual knowledge. As a result of the experiment, we present that the FoCus model could not correctly blend the knowledge according to the input dialogue and that the dataset design is unsuitable for the multi-turn conversation.

## introduction recent studies have widely considered knowledge grounded dialogue, user interest, and preference @xcite @xcite @xcite @xcite @xcite . especially, @xcite presented persona-chat dataset based on personalizing dialogue agents. similarly, @xcite constructed conversation dataset with emotional labels according to the given situation. in different way, @xcite and @xcite focused on generating dialogue based on knowledge retrieved from wikipedia. however, those existing dialogue datasets do not comprehensively consider the user persona of the given situation and knowledge of the grounded object. the biased dataset toward persona or knowledge quickly undermines the user's intention or purpose, and rendering high-quality answers is challenging. from this point of view, dialogue generation through the proper blending of persona and knowledge is a significant issue that should be considered in advanced research. to fulfill this purpose, @xcite has released focus and baseline models based on the grounded persona and knowledge. to the best of our knowledge, focus is the first persona-knowledge grounded dialogue dataset that incorporates knowledge and customized persona. however, related research in the personaknowledge is still insufficient, and @xcite only provides descriptions of the dataset and has not corroborated the weakness of the dataset and model architecture. furthermore, no in-depth analysis has been conducted on whether the model engages in conversation based on the blended personaknowledge. to demonstrate these problems, we intend to conduct an in-depth analysis of the proposed model and dataset through an empirical study. in this paper, we execute probing tests by throwing research questions in terms of data-centric @xcite and model-centric @xcite in the @xcite . first, we point out that focus is more of a question-answering than a multi-turn dialogue task, under the question "is the focus whether for conversation or question answering?" generally, a multi-turn dialogue dataset forms a context in which two or more speakers continue to conversation. moreover, the memorable context significantly impacts the generation of the next utterance. however, we experimentally found that the previous conversation on the focus had little effect on the next utterance. this is because each round consists of an independent set that is not involved in the context. therefore, we demonstrate that the focus is more of a question-answering task rather than a consistent multi-turn conversation. to closely analyze this point, we conduct a case study according to the change in the conversation order and the inclusion of dialogue histories corresponding to the previous utterance. second, we ask, "does the focus model do real knowledge blending?" and examine whether the baseline model presented in @xcite is blending properly with persona in selecting the appropriate knowledge for conversation. in this experiment, we proceed with various analyses using bm25 @xcite , dpr @xcite , sts @xcite , and tf-idf @xcite as a retrieval module. we analyze performance changes according to the knowledge selection of each search module. based on this, we attempt to probe what problems the method of blending history and selected knowledge in an in-context approach causes in knowledge grounding. ## conclusion in this work, we conduct an in-depth analysis of focus, which aims to blend knowledge and persona. experimental results quantitatively demonstrate that the proposed content as a multi-turn dialogue is close to qa and that the model does not appropriately incorporate knowledge to persona. in the future, we plan to properly combine knowledge and persona based on the limitations we presented.