Anticipating Follow-Up Questions in Exploratory Information Search (2024)

Abstract: The paper describes methods for anticipating follow-up questions in exploratory information search. There are two main cases: information stored in knowledge graphs, and information in unstructured texts such as Wikipedia. In the first case, follow-up questions are anticipated by extracting subgraphs relevant to user queries, passing the subgraphs to an LLM to generate responses. In the second case, entities and their relationships are extracted from the texts and added to short-term knowledge graphs relevant to initial queries. Follow-up questions are then anticipated by extracting subgraphs relevant to subsequent queries and passing the subgraphs to the LLM, as in the first case. The short-term graphs in dialogue memory are often sufficient to answer follow-up questions. If they are not, the described steps are repeated as required.

## introduction dialogue systems that support users in exploratory information search typically need to handle many follow-up questions. the paper describes methods for anticipating follow-up questions in dialogues for exploratory information search. there are two cases: exploring information stored in knowledge graphs, and exploring information in unstructured texts such as wikipedia. the dialogues are exploratory because the users do not yet know where the information is located, or even if it exists. they may not know the structure of the knowledge graphs, or what taxonomy has been used to classify the information into different categories. as a result, users need to keep asking questions as they learn to navigate around different information spaces. the proposed approach aims to anticipate likely follow-up questions by constructing subgraphs of entities and relationships relevant to current and recent user queries. this can be done while the user is thinking what question to ask next. if a user is searching existing knowledge graphs, likely follow-up questions can be anticipated by extracting subgraphs relevant to the current user query. the subgraphs are included in prompts to llms to generate responses to the user. if a user is searching unstructured texts such as wikipedia, there is no knowledge graph from which subgraphs can be extracted. in this case an llm is prompted to extract entities from the user query, and to extract relevant entities and relationships from the texts, and finally to construct a small shortterm knowledge graph from them. the paper is structured as follows. section 2 discusses related work. section 3 summarizes existing methods for generating natural language responses from wikipedia texts and from knowledge graphs. section 4 describes new methods for generating subgraphs from existing knowledge graphs and for generating new knowledge graphs from texts. in section 5 the new methods are used to anticipate follow-up questions in a hybrid retrieval approach combining structured and unstructured retrieval. a system combining conversational agents with knowledge graphs in neo4j databases is described by @xcite . a similar system from @xcite aims for synergy between knowledge graphs and conversational agents by bridging the gap between structured and unstructured information retrieval, a topic also addressed here in section 5 on hybrid retrieval. concerning methods for anticipating follow-up questions in exploratory search, schneider et al. (2023b) mention wikitalk @xcite , an early robot dialogue system for exploratory search in wikipedia. using no knowledge graphs, wiki-talk extracted sets of hyperlinks from wikipedia articles to transition smoothly between topics by anticipating what the user will ask about next. @xcite proposed a method for anticipation of follow-up topics in wikipedia search based on hyperlinks and keywords extracted from the current article. this enables anticipating follow-up topics that have no explicit link, and also works for documents without hyperlinks. the wikitalk approach of extracting small sets of wikipedia hyperlinks from the current topic to related topics was motivated by the need at that time to restrict speech recognition vocabulary to a finite list of predicted phrases @xcite . however, the basic idea is similar to retrieving a subgraph or neighborhood of relevant nodes from a knowledge graph, as described in section 4.1. rag @xcite is often described as a way of grounding llm responses in the retrieved information, but conversational grounding has a long history in dialogue systems research @xcite . grounding is especially important in open-ended conversational exploratory search for navigation in unknown information landscapes @xcite . theory of mind errors often arise from failure to build shared knowledge during the dialogue @xcite . @xcite investigate the capacity of llms to build shared knowledge by classifying grounding-related dialogue acts and by extracting mutually grounded information. ## llms that generate responses rag enables llms to generate natural language responses from retrieved information that is not in their training corpora. this section compares existing methods for rag from wikipedia texts and rag from knowledge graphs. ## llms that generate graphs we now describe methods for generating subgraphs from existing knowledge graphs and for generating new knowledge graphs from texts. ## anticipating follow-up questions in the hybrid retrieval approach has been applied to anticipating follow-up questions in exploratory information search from wikipedia. this short paper does not include evaluation and analysis sections. however, the hybrid retrieval approach can be compared with the simple rag from wikipedia texts described in section 3.1. in a small trial, llmgraphtransformer was used to construct a knowledge graph about william shakespeare from wikipedia texts. appendix a lists 50 of the generated relationships. there are some technical problems in generating knowledge graphs using llms. one problem is getting the direction of relationships correct. for example in ## conclusion after summarizing existing methods for generating natural language responses from wikipedia texts and from knowledge graphs, the paper described new methods for anticipating follow-up questions in dialogues for exploratory information search, considering two cases. when exploring information already stored in knowledge graphs, follow-up questions are anticipated by extracting subgraphs that are likely to be relevant to subsequent user queries, and passing the subgraphs to an llm to generate responses. when exploring information in unstructured texts such as wikipedia, entities and relationships are extracted from the texts and used to construct new short-term knowledge graphs relevant to initial user queries. follow-up questions are anticipated by extracting subgraphs likely to be relevant to subsequent user queries, and continuing as in the first case. although there are some problems to be solved in automatic construction of knowledge graphs by llms, this kind of approach is attractive. ongoing work will aim to explore its potential benefits both for anticipating follow-up questions in exploratory information search, and more widely in other areas of spoken dialogue systems.