Information Seeking in the Spirit of Learning: A Dataset for Conversational Curiosity (2020)

Abstract: Open-ended human learning and information-seeking are increasingly mediated by digital assistants. However, such systems often ignore the user’s pre-existing knowledge. Assuming a correlation between engagement and user responses such as “liking” messages or asking followup questions, we design a Wizard-of-Oz dialog task that tests the hypothesis that engagement increases when users are presented with facts related to what they know. Through crowd-sourcing of this experiment, we collect and release 14K dialogs (181K utterances) where users and assistants converse about geographic topics like geopolitical entities and locations. This dataset is annotated with pre-existing user knowledge, message-level dialog acts, grounding to Wikipedia, and user reactions to messages. Responses using a user’s prior knowledge increase engagement. We incorporate this knowledge into a multi-task model that reproduces human assistant policies and improves over a bert content model by 13 mean reciprocal rank points.

## introduction conversational agents such as alexa, siri, and google assistant should help users discover, learn, and retain novel factual information. more generally, systems for conversational informationseeking should help users develop their information need, be mixed-initiative, incorporate user memory, and reason about the utility of retrieved information as a combined set @xcite . we focus on a curiosity-driven, informationseeking scenario where a user starts a conversation with an assistant by asking an open-ended question and then drills down into interest areas (figure 1 ). in this setting, what policies should assistants pursue to maintain the user's interest in the topic? theories of human learning, such as vygotsky's zone of proximal development, propose that learning novel information should be rooted in preexisting knowledge and skills of the learner @xcite . considering this, a good policy may give general information about tahiti; a better policy would select information related to the user's knowledge (e.g., familiarity with france). we hypothesize that engagement is correlated with policies that integrate a user's pre-existing knowledge, and test this through a large-scale, wizardof-oz (woz) style collection @xcite that captures assistant policies, user reactions, and topically relevant entities that the user knows about. the curiosity dataset has 14,048 english dialogs annotated with sentence-level knowledge grounding, the user's prior knowledge, dialog acts per message, and binary ratings per message. 1 in our dialog task (figure 2 ), one worker takes the role of a curious user learning about a geographic entity and the other of a digital assistant with access to wikipedia facts (section 2). at the start of each dialog, the user is assigned an entity as their topic (e.g., puerto rico) along with two aspects (e.g., history and demographics) to investigate. beforehand, we show the user a list of entities related to the topic, and they mark which they know; these entities are a sample of their preexisting knowledge. the user engages in openended discovery while the assistant simultaneously answers the user's questions and proactively introducing facts likely to prompt followup questions. section 3 uses dialog act annotations combined with explicit and implicit user feedback to compare assistants' content selection and presentation policies. for example, in interactions where the user asks a question and the assistant paraphrases a fact, how often does the user ask a followup question versus trail off in disinterest? most datasets (section 6) do not have enough annotations to answer these questions: it requires message-level dialog act annotations and feedback signals. we compare three assistant policies: using a fact with a rooted entity, a fact from the user's aspect, or a generic fact about the topic. the policies are compared through user "likes" of assistant messages and by the dialog act of their subsequent message (e.g., did they ask a specific followup or change topic). in section 4, we design models that predict the policies used by the assistant: what type of message to send and which fact to use (if any). all models are trained jointly with a multi-task objective function. we compare an end-to-end bert @xcite model to our task-specific hierarchical recurrent encoder model @xcite and show that our model improves over the baseline. in summary, we make three main contributions: (1) we design an experiment to test the efficacy of personalizing conversational information systems through a user's prior knowledge, (2) introduce the curiosity dataset-the first dialog dataset combining sentence-level knowledge groundings, per message ratings, and per message dialog act annotations, allowing for robust and fine-grained structural learning of dialog policies for similar applications, and (3) design a multi-task model that incorporates the user's prior knowledge and improves over a natural bert baseline. ## building the curiosity dataset this section describes the construction of the curiosity dataset. dialog topics consist of prominent world geographic entities. the worldwide spread of entities makes each novel to most users, the consistent topic type makes starting dialogs easier, and their rich histories, demographics, and economics add topical diversity. for example, most people are only vaguely familiar with the history of puerto rico, but most know about related concepts such as the united states or hurricane maria. section 2.1 describes how we select geographic topics, aspects, and derive a set of facts to ground against. we collected the dataset in two steps: (1) collecting dialogs with a custom interface (section 2.2) and (2) after-the-fact dialog act annotation (section 2.3). sample dialogs from curiosity are in appendix c. ## dataset analysis this section shows statistics of the curiosity dataset and that users prefer aspect-specific, rooted facts. ## models we design a machine learning model that predicts assistant and user actions. we introduce a multitask architecture for curiosity that hierarchically models (charm, figure 4 ) dialogs to: (1) predict the dialog acts of the user message (utterance act prediction), (2) select the best fact (fact prediction), (3) choose the best set of dialog acts for the next message (policy act prediction), and (4) predict if the assistant message will be liked (like prediction). ## modeling experiments charm improves over a bert model in most tasks. ## related work our work builds on knowledge-grounded conversational datasets and modeling. datasets although there are numerous grounded datasets, we did not find one for conversational information seeking that contained fine-grained knowledge groundings, message-level feedback from the user, and dialog acts. topical chat @xcite search as a conversation @xcite wizard of wikipedia @xcite quac @xcite cmu dog @xcite ms marco conv. @xcite n/a n/a n/a n/a opendialkg @xcite coqa @xcite holl-e @xcite commonsense @xcite reddit+wiki @xcite our dataset is most similar to those for information-seeking such as quac @xcite , wizard of wikipedia @xcite , cmu dog @xcite , ms marco @xcite , topical chat @xcite , the trec conversational assistance track @xcite , and search as a conversation @xcite . quac constrains assistant responses to spans from wikipedia, which makes it better for conversational question answering, but prevents more sophisticated assistant policies. quac also provides dialog acts, but they exist so that the assistant can inform the user of valid actions; we annotate dialog acts after-the-fact so that we can compare freely chosen user responses. like quac, topical chat, saac, and wow have annotated knowledge-groundings for each message, but responses are free-form. saac is a contemporaneous, cast-based dataset that shares our motivation to make conversation a medium for informationseeking. topical chat includes user feedback, but instead of explicitly defined roles, workers implicitly take dual and alternating roles as the user and assistant through knowledge asymmetry; followup work added automatically annotated dialog acts to topical chat @xcite . many tasks instruct annotators to take on a specific role in the dialog. for example, in wizard of wikipedia, annotators assume an assigned persona @xcite in addition to being the user or assistant. consequently, many dialogs revolve around personal discussions instead of teaching about a topic. additionally, annotators may not have the background to play their role. in contrast, we ask annotators to take roles that-as humansthey already know how to do: read about and convey interesting information on a topic (assistant) and engage in inquiry about a novel topic (user). our work is one of many in knowledge-grounded conversational datasets. for example, @xcite have workers discuss movies and ground messages to plot descriptions, reviews, comments, and factoids; however, one worker plays both roles. in opendialkg @xcite , annotators ground messages by path-finding through freebase @xcite while discussing and recommending movies, books, sports, and music. @xcite use reddit discussion threads as conversations and ground to web pages. similarly, @xcite collect twitter threeturn threads and ground to foursquare restaurant reviews. our work adds to this dataset compendium. external knowledge in models our model is related to those that incorporate external information like facts in question answering @xcite @xcite , knowledge base triples in dialog models @xcite @xcite , common sense @xcite , or task-specific knowledge @xcite . similarly to @xcite ; @xcite , charm predicts the act of the current message, but also next message's act like @xcite do. ## future work and conclusion we see two immediate directions for future work. the first is to augment our charm model with a text generation module to make a digital version of our human assistants. this involves contextualizing and paraphrasing facts which our dataset supports. second, dialog act sequences could identify additional data-driven policies that could be used to define rewards or losses. by conditioning on dialog acts or sequences of dialog acts, textual outputs could be better-controlled @xcite and combined with knowledge grounding @xcite . however, text is not the native modality of digital assistants. we envision digital assistants participating in information-seeking, which means handling speech input. consequently, automatic speech recognition (asr) introduces transcription errors which are especially prevalent in knowledge-oriented text like question answering @xcite . @xcite show this is also problematic in information-seeking dialog by comparing models on textual and asr versions of topical chat. to close the loop in conversational informationseeking, models need to account for the speechbased environment of digital assistants. in summary, this work introduces curiosity: a large-scale conversational information seeking dataset. with curiosity's unique set of annotations, we design charm which jointly learns to choose facts, predict a policy for the next message, classify dialog acts of messages, and predict if a message will be liked. we hope that our dataset will encourage further interest in curiosity-driven dialog.