What’s The Latest? A Question-driven News Chatbot (2020)

Abstract: This work describes an automatic news chatbot that draws content from a diverse set of news articles and creates conversations with a user about the news. Key components of the system include the automatic organization of news articles into topical chatrooms, integration of automatically generated questions into the conversation, and a novel method for choosing which questions to present which avoids repetitive suggestions. We describe the algorithmic framework and present the results of a usability study that shows that news readers using the system successfully engage in multi-turn conversations about specific news stories.

## introduction chatbots offer the ability for interactive information access, which could be of great value in the news domain. as a user reads through news content, interaction could enable them to ask clarifying questions and go in depth on selected subjects. current news chatbots have minimal capabilities, with content hand-crafted by members of news organizations, and cannot accept free-form questions. to address this need, we design a new approach to interacting with large news collections. we designed, built, and evaluated a fully automated news chatbot that bases its content on a stream of news articles from a diverse set of english news sources. this in itself is a novel contribution. our second contribution is with respect to the scoping of the chatbot conversation. the system organizes the news articles into chatrooms, each revolving around a story, which is a set of automatically grouped news articles about a topic (e.g., articles related to brexit). the third contribution is a method to keep track of the state of the conversation to avoid repetition of information. for each news story, we first generate a set of essential questions and link each question with content that answers it. the motivating idea is: two pieces of content are redundant if they answer the same questions. as the user reads content, the system tracks which questions are answered (directly or indirectly) with the content read so far, and which remain unanswered. we evaluate the system through a usability study. the remainder of this paper is structured as follows. section 2 describes the system and the content sources, section 3 describes the algorithm for keeping track of the conversation state, section 4 provides the results of a usability study evaluation and section 5 presents relevant prior work. the system is publicly available at https:// newslens.berkeley.edu/ and a demonstration video is available at this link: https://www. youtube.com/watch?@xmath0. ## system description this section describes the components of the chatbot: the content source, the user interface, the supported user actions and the computed system answers. appendix a lists library and data resources used in the system. ## conversation state one key problem in dialogue systems is that of keeping track of conveyed information, and avoiding repetition in system replies (see example in figure 2 ). this problem is amplified in the news setting, where different news organizations cover content redundantly. we propose a solution that takes advantage of a question and answer (q&a) system. as noted above, the motivating idea is that two pieces of content are redundant if they answer the same questions. in the example of figure 2 , both system messages answer the same set of questions, namely: "when did the fires start?", "how many people have died?" and "how many hectares have burned?", and can therefore be considered redundant. our procedure to track the knowledge state of a news conversation consists of the following steps: (1) generate candidate questions spanning the knowledge in the story, (2) build a graph connecting paragraphs with questions they answer, (3) during a conversation, use the graph to track what questions have been answered already, and avoid using paragraphs that do not answer new questions. question candidate generation. we fine-tune a gpt2 language model @xcite on the task of question generation using the squad 2.0 dataset @xcite . at training, the model reads a paragraph from the training set, and learns to generate a question associated with the paragraph. for each paragraph in each article of the story (the paragraph set), we use beam search to generate k candidate questions. in our experience, using a large beam size (@xmath1, as one paragraph can yield several valid questions. beam search enforces exploration, with the first step of beam search often containing several interrogative words (what, where...). for a given paragraph, we reduce the set of questions by deduplicating questions that are lexically close (differ by at most 2 words), and removing questions that are too long (>12 words) or too short (<5 words). building the p/q graph. we train a standard q&a model, a roberta model @xcite finetuned on squad 2.0 @xcite , and use this model to build a paragraph / question bipartite graph (p/q graph). in the p/q graph, we connect any paragraph (p node), with a question (q node), if the q&a model is confident that paragraph p answers question q. an example bipartite graph obtained is illustrated in figure 3 , with the question set on the left, the paragraph set on the right, and edges between them representing model confidence about the answer. because we used a large beam-size when generating the questions, we perform a pruning step on the questions set. our pruning procedure is based on the realization that two questions are redundant if they connect to the same subset of paragraphs (they cover the same content). our objective is to find the smallest set of questions that cover all paragraphs. this problem can be formulated as a standard graph theory problem known as the set cover problem, and we use a standard heuristic algorithm @xcite . after pruning, we obtain a final p/q graph, a subgraph of the original consisting only of the covering set questions. the p/q graph embodies interesting properties. first, the degree of a question node measures how often a question is answered by distinct paragraphs, providing a measure of the question's importance to the story. the degree of a paragraph node indicates how many distinct questions it answers, an estimate of its relevance to a potential reader. finally, the graph can be used to measure question relatedness: if two questions have non-empty neighboring sets (i.e., some paragraphs answer both questions), they are likely to be related questions, which can be used as a way to suggest follow-up questions. using the p/q graph. at the start of a conversation, no question is answered, since no paragraph has been shown to the user. therefore, the system initializes a blank p/q graph (left graph in figure 3 ). as the system reveals paragraphs in the conversation, they are marked as read in the p/q graph (shaded blue paragraphs in the right graph of figure 3 ). according to our q&a model, any question connected to a read paragraph is answered, so we mark all neighbors of read paragraphs as answered questions (shaded blue questions on the right graph of figure 3 ). at any stage in the conversation, if a paragraph is connected to only answered questions, it is deemed uninformative, as it will not reveal the answer to a new question. as the conversation moves along, more paragraphs are read, increasing the number of answered questions, which in turn, increases the number of uninformative paragraphs. we program the system to prioritize paragraphs that answer the most unanswered questions, and disregard uninformative paragraphs. we further use the p/q graph to recommend questions to the user. we select unanswered questions and prioritize questions connected to more unread paragraphs, recommending questions three at a time. ## study results we conducted a usability study in which participants were assigned randomly to one of three configurations: these are contrasted in order to test (a) if showing automatically generated questions is beneficial to news readers, and (b) to assess the question tracking algorithm against a similar question recommendation method with no conversation state. ## discussion during the usability study, we obtained direct and indirect feedback from our users, and we summarize limitations that could be addressed in the system. inability to handle small talk. 4 participants attempted to have small talk with the chatbot (e.g. asking "how are you"). the system most often responded inadequately, saying it did not understand the request. future work may include gently directing users who engage in small talk to a chitchat-style interface. inaccurate q&a system. 32% of the participants mentioned that answers are often off-track or irrelevant. this suggests that further improvements in q&a systems are needed. dealing with errors. within the current framework, errors are bound to happen, and easing the user's path to recovery could improve the user experience. ## conclusion we presented a fully automated news chatbot system, which leverages an average of 2,000 news articles a day from a diverse set of sources to build chatrooms for important news stories. in each room, the system takes note of generated questions that have already been answered, to minimize repetition of information to the news reader. a usability study reveals that when the chatbot recommends questions, news readers tend to have longer conversations, with an average of 24 messages exchanged. these conversation consist of combination of recommended and user-created questions.