Multi-Sentence Knowledge Selection in Open-Domain Dialogue (2021)

Abstract: Incorporating external knowledge sources effectively in conversations is a longstanding problem in open-domain dialogue research. The existing literature on open-domain knowledge selection is limited and makes certain brittle assumptions on knowledge sources to simplify the overall task, such as the existence of a single relevant knowledge sentence per context. In this work, we evaluate the existing state of open-domain conversation knowledge selection, showing where the existing methodologies regarding data and evaluation are flawed. We then improve on them by proposing a new framework for collecting relevant knowledge, and create an augmented dataset based on the Wizard of Wikipedia (WOW) corpus, which we call WOW++. WOW++ averages 8 relevant knowledge sentences per dialogue context, embracing the inherent ambiguity of open-domain dialogue knowledge selection. We then benchmark various knowledge ranking algorithms on this augmented dataset with both intrinsic evaluation and extrinsic measures of response quality, showing that neural rerankers that use WOW++ can outperform rankers trained on standard datasets.

## introduction one of the key components needed to enable robust and engaging open-domain conversational systems is the ability to select and integrate relevant knowledge from diverse knowledge sources. this knowledge selection setting is complex for a number of reasons: 1) relevant knowledge is highly dependent on conversational context, and requires understanding dialogue history and evolving user requirements at a turn-by-turn granularity, 2) because conversations are not topic-constrained, systems may need to pool knowledge from a theoretically boundless number of sources (i.e. the entire figure 1 : overview of an end-to-end knowledge ranking and response generation system internet), and 3) knowledge may be represented in structured, semi-structured, and unstructured formats making it difficult to extract all the information needed for a conversation. recently there has been increasing interest in the problem of knowledge selection for open-domain dialogue @xcite @xcite . there have been numerous efforts to build standardized benchmark datasets whereby open-domain conversations are collected with crowdworkers who are given access to some closed set of knowledge sources such as a subset of wikipedia or news articles @xcite . these datasets suffer from a number of limitations including the fact that they either do not explicitly annotate relevant knowledge sentences per turn in the conversation or make the strict assumption that only a single utterance from a knowledge source is relevant. without sufficient information about relevant knowledge for a given context, it is difficult to train data-driven models on the isolated problem of knowledge selection. in this work, we conduct a thorough analysis of the knowledge selection problem through the lens of the wizard of wikipedia (wow) dataset, one of the standard knowledge-grounded open-domain dialogue benchmarks @xcite . our analysis qualitatively and quantitatively demonstrates that the strict one-knowledge-sentence-forone-context assumption in the data is unreasonable, leading to a lack of meaningful interannotator agreement scores. we then build on this result and relax this assumption, allowing for multiple knowledge snippets per context. we introduce a new continuouslyscaled notion of relevance called wisdom-of-thecrowd-relevance (woc) and use this measure to reannotate about 800 dialog contexts from the original wow corpus with relevant knowledge. this is done by taking a dialogue from wow and extracting a subdialogue at a random turn in the dialogue. our augmented wow corpus, which we call wow++, averages 8 knowledge sentences per dialogue turn, and demonstrates significantly more knowledge diversity. using wow++, we then benchmark a number of different knowledge ranking algorithms using both standard information retrieval automatic measures as well as extrinsic human evaluation on generated responses. our results indicate that neural rerankers using wow++ are able to outperform other algorithms such as traditional ir baselines and neural models trained using the original wow data. ## related work in recent years, knowledge selection in opendomain dialogue systems has seen a tremendous surge in interest as the community has recognized the utility of having these abilities in conversational systems @xcite @xcite . in one line of work, a number of industry research groups have demonstrated that large quantities of chat data coupled with the latest highcapacity transformer-based models can produce particularly engaging and convincing conversational experiences @xcite . while these models produce impressive outputs, they consciously shirk any explicit knowledge-selection mechanisms. any knowledgeable appearance in their outputs tends to be a consequence of facts memorized in training data @xcite . in addition, the models have a tendency to generate facts that may be factually inaccurate, referred to as factual hallucination. knowledge selection in open-domain systems took a tremendous leap forward with the introduction of standard datasets of knowledge-grounded dialogues. the work of @xcite @xcite produced such corpora with upwards of 10k dialogues and up to 10s of dialogue turns leveraging knowledge from diverse sources such as wikipedia, and the washington post. while certainly a step forward, these datasets introduced some unreasonable data constraints that aren't apt to the knowledge setting such as either no explicitly-annotated knowledge snippets or only a single one, making training of robust knowledge selection systems very difficult. since the introduction of these corpora, numerous groups have tackled the knowledge selection problem from different angles. for example, @xcite developed a sequential latent variable model to help address ambiguity in knowledge sources in the wow context. @xcite developed models that dealt with low-resource settings with general representations learned from ungrounded dialogues but finetuning done with small numbers of domain-specific training examples. @xcite recognized that even with external knowledge sources, there may be a knowledge gap that can be filled in real-time using unsupervised local knowledge. while these works created modeling improvements on existing datasets, there has still not been a study investigating how well-formed our existing datasets are. ## wow++ the wow++ dataset we describe below is an augmented dataset based on the wizard of wikipedia (wow) corpus @xcite . the wow corpus consists of 22,311 dialogues containing 201,999 turns. the dialogues are comprised of two interlocutors who engage in chit chat on a given topic where one interlocutor is a knowledgeable expert in the topic. the expert, or wizard, is provided access to knowledge snippets from wikipedia that are potentially relevant to a given conversation topic, asked to select one of the knowledge snippets and utilize the information from the knowledge snippet in their response. thus, for a given wizard utterance only a single knowledge snippet is selected from a set of potentially relevant knowledge snippets. this selected snippet is considered to be the ground-truth, referred to as the gold sentence, and is referred to as such throughout this paper. human: [...] do you know any famous guitar players, or have a favorite guitarist that you listen to? original wow ground-truth knowledge snippet: ¨ eric patrick clapton (born 1945) is an english rock and blues guitarist singer and songwriter. alternative relevant knowledge snippets: ¨ george buddy guy (born july 30 1936) is an american blues guitarist and singer. ¨ doyle bramhall ii (born 24 december 1968) is an american musician, producer, guitarist, and songwriter known for his work with eric clapton, roger waters, and many others. ¨ john mayall obe (born 29 november 1933) is an english blues singer, guitarist, organist, and songwriter whose musical career spans over fifty years. ## method we evaluate wow++ using two different regimes to see the effect of the augmented knowledge snippets. the first is intrinsic evaluation for the knowledge reranking or selection task using automatic measures. the second is extrinsic evaluation where we provide a selected knowledge candidate to a response generation model and perform human annotation of system generated responses. ## conclusion in this work, we demonstrated that knowledge selection is an intrinsically ambiguous task for open-domain conversations, which necessitates improved notions of relevance in our benchmark datasets. we used this insight to propose a new measure for knowledge sentence relevance called wisdom-of-the-crowd relevance. using this measure, we annotated a new collection of dialogues with relevant knowledge called wow++ (it will be released publicly). we then evaluated a number of knowledge selection algorithms on our new dataset using both intrinsic and extrinsic metrics, and demonstrate that neural rankers trained leveraging wow++ can outperform traditional knowledge selection algorithms. it is worth noting, however, that annotating a knowledge selection dataset with all relevant snippets as we have done for wow++ is a time-intensive task that may be expensive to scale up. future work should investigate how to develop more low-resource rankers or how to bootstrap from a high quality seed dataset like wow++ to a larger corpus.