Getting Better Dialogue Context for Knowledge Identification by Leveraging Document-level Topic Shift (2022)

Abstract: To build a goal-oriented dialogue system that can generate responses given a knowledge base, identifying the relevant pieces of information to be grounded in is vital. When the number of documents in the knowledge base is large, retrieval approaches are typically used to identify the top relevant documents. However, most prior work simply uses an entire dialogue history to guide retrieval, rather than exploiting a dialogue’s topical structure. In this work, we examine the importance of building the proper contextualized dialogue history when document-level topic shifts are present. Our results suggest that excluding irrelevant turns from the dialogue history (e.g., excluding turns not grounded in the same document as the current turn) leads to better retrieval results. We also propose a cascading approach utilizing the topical nature of a knowledge-grounded conversation to further manipulate the dialogue history used as input to the retrieval models.

## introduction knowledge identification (ki) is the task of identifying relevant information from a database of documents that should be used when generating responses in a knowledge-grounded dialogue system @xcite . when the number of documents is large, information retrieval is typically used to find relevant documents @xcite @xcite . most approaches encode both the knowledge sources and the dialogue context (i.e., all prior turns), which is later used as an input query, into the same vector space. since the quality of the input query significantly impacts the retrieval results @xcite , using an optimal dialogue context is crucial. in knowledge-grounded dialogues, each turn can be grounded in a different document. blindly including all previous turns into the dialogue context can introduce unnecessary noise because a turn grounded in a different document can provide redundancy or irrelevant information for the grounding process of the current turn. our hypothesis is that including only turns in the dialogue context that are grounded in the same document as the current turn when creating a retrieval query will improve ki task performance. to test this hypothesis, we tried several approaches to select relevant turns to be included in the dialogue context. specifically, we vary the input to a previously used predictive model @xcite to see whether querying using only turns grounded in the same document as the current turn improves retrieval performance. after verifying our hypothesis using oracle results, we utilize automatically computed document-level topic shifts to improve the dialogue context used for ki. even with imperfect automatic predictive models, our initial results show that improving dialogue context increases the retrieval results on dialogues grounded on at least 2 documents. further analysis on errors from dialogues grounded only in 1 document leads us to a simple heuristic that raises the retrieval accuracy for the entire dataset. our contribution is twofold. first, we verify the importance of a proper contextualized query in the ki task, as excluding utterances from the dialogue context that are not grounded in the same document as the current turn leads to better knowledge retrieval results in an oracle condition. second, based on that verification, we develop a simple automatic approach that improves ki in document-grounded dialogue by leveraging a proposed topic segmentation algorithm that uses both dialogue content and grounding documents. ## related work our work is related to recent work in knowledge identification (ki) in knowledge-grounded dialogues @xcite @xcite @xcite . however, prior work has largely figure 1 : an example dialogue from multidoc2dial @xcite ) that is grounded in three different documents. our focus on document-level topic shifts in dialogue is related to the task of discourse segmentation. prior work in identifying topic changes has used topic tracking with predefined topics @xcite and used coherence scores between consecutive utterances to split the conversation into smaller topics @xcite . however, such segmentation approaches have typically been based solely on the content of conversations. in contrast, we propose a topic segmentation approach based not only on the dialogue content, but also on the grounding document. ## background passages as retrieval units. since a grounding document can be very long, we split each one into passages and use them as the units for retrieval. we follow @xcite to split a document based on its original paragraphs indicated by markup tags and then attach the hierarchical titles from their html source to each paragraph as a passage. ## method: document-level topic shift since ki methods typically use all previous turns as the dialogue context, instead of focusing on improving model architectures for knowledge-grounded response generation, we examine whether varying the input (e.g., dialogue context) to such models improves the retrieval and generation results. specifically, we hypothesise that for the current turn t i , including only previous turns grounded in the same document as t i in the dialogue context to dpr will improve the overall passage retrieval results. to verify this hypothesis, we first create an oracle model called rag-oracle. it assumes that the correct grounding passages of previous turns are known, so it only uses the turns grounded in the same document as t i in the input query to dpr. however, since the gold-standard grounding information of the dialogue is not available in real use cases, we build a simple classification model to estimate it. this model, which we call the passage checking model (pc), is a bert model finetuned on multidoc2dial. the input includes the current user turn t i , the dialogue context h, and one passage p. the output is 1 if t i should be grounded in p given h and 0 otherwise. during training, the dialogue context only contains turns grounded in the same document as t i . for each training instance, we sample 128 negative passages 1 , at most half of them are from the same document which p belongs to and the rest are from different documents. our pc model achieved 69.4 f 1 score on validation set. we also use the probability scores from the last layer (softmax) as a confidence measure below. next, we use pc in a cascading algorithm to retrieve the top 10 passages for the current user turn (details in algorithm 1). for each conversation, we process the turns increasingly while keeping track of a list of documents (docs) that have been used for grounding so far. at each turn t i , we try to ground it to each document in docs and use only turns grounded in the same targeting document as the dialogue context. we add the documents containing one of the top-10 passages to the set docs before going to the next turn. the model based on this algorithm is called rag-cascade. finally, since the bart generator relies on the top-5 passages to provide the grounding span, having a better top-5 can yield improved generation results. we explore this idea by reusing the probability scores from the pc model as a ranking metric instead of building another ranking model. @xcite , all numbers reported in this section are the mean of three runs with different random seeds. for retrieval, we use recall at k (r@k), which measures the frequency of the correct passage found in the top-k retrieved passages. token-level f 1 score and exact match (em) @xcite are used to evaluate the grounding span generation results. implementation details can be found in appendix a. ## conclusion in this work, we showed that exploiting documentlevel topic shifts in document-grounded dialogues relying on multiple documents as the knowledge base can raise passage retrieval results. we first proposed a simple cascading approach based on a simple bert model for passage checking and re-ranking that yielded improved retrieval results for multiple-segment dialogues. an error analysis suggested that limiting the number of turns in the dialogue context to 6 reduced the false segmentation errors for the one-segment dialogues and thus improved the scores for the full corpus. furthermore, no improvement from span generation with the increased retrieval results implied that a general-purpose generative model like rag might not be a good fit for knowledge identification task in information-seeking dialogues. future plans include using better generative models to generate better system responses from the identified knowledge and conducting further analysis on the segmentation yielded from the proposed algorithm.