Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations (2024)

Abstract: While language model (LM)-powered chatbots and generative search engines excel at answering concrete queries, discovering information in the terrain of unknown unknowns remains challenging for users. To emulate the common educational scenario where children/students learn by listening to and participating in conversations of their parents/teachers, we create Collaborative STORM (Co-STORM). Unlike QA systems that require users to ask all the questions, Co-STORM lets users observe and occasionally steer the discourse among several LM agents. The agents ask questions on the user’s behalf, allowing the user to discover unknown unknowns serendipitously. To facilitate user interaction, Co-STORM assists users in tracking the discourse by organizing the uncovered information into a dynamic mind map, ultimately generating a comprehensive report as takeaways. For automatic evaluation, we construct the WildSeek dataset by collecting real information-seeking records with user goals. Co-STORM outperforms baseline methods on both discourse trace and report quality. In a further human evaluation, 70% of participants prefer Co-STORM over a search engine, and 78% favor it over a RAG chatbot.

## introduction recent advancements in language models (lms) @xcite ; openai, 2023; gemini team, 2024) and retrieval-augmented generation (rag) @xcite have led to more capable chatbots and emerging generative search engines @xcite . compared to traditional search engines and information retrieval (ir) models @xcite , these systems fulfill user queries by generating direct responses, effectively addressing known unknowns, where users are aware of their information needs. however, a gap remains in using these systems for complex information-seeking scenarios, such as academic research, market analysis, and decisionmaking, where the system should expose users to their unknown unknowns to facilitate knowledge discovery. while the concept of unknown unknowns originally referred to unexpected risks in the military, it is linked to the serendipitous discovery of information in the information research context @xcite . specifically, kirzner (1997) directly contrasts such discovery ("the realization that one had overlooked something in fact readily available") with successful search ("the deliberate production of information which one knew one had lacked"). prior work on automated expository writing attempts to help readers navigate the terrain of unknown unknowns by curating information from various sources into unified articles with substantial breadth and depth @xcite . in particular, the storm writing system demonstrates that lms paired with search engines can automatically generate a high-quality draft of wikipedia-like articles on arbitrary topics @xcite . however, producing the static report as the final outcome, storm does not support any user interaction which is crucial in complex information seeking where there is no single, gold query, but queries evolve dynamically towards a goal @xcite . this dynamic and exploratory nature makes designing assistance systems challenging. traditional search engines and rag chatbots passively react to users' search queries or conversational questions, often inducing echo chamber effects @xcite or high cognition load as users with limited prior knowledge may even struggle to formulate questions @xcite . to surface unknown unknowns and better support user interaction, we propose collaborative storm (co-storm), an information-seeking assistance system that supports collaborative discourse among users and multiple lm agents. unlike the one-question-one-answer mode of interaction, co-storm allows users to learn by observing and participating occasionally in the discourse, emulating a common educational scenario @xcite . to facilitate a thought-provoking discourse and serendipitous discovery, co-storm simulates two agent types grounded in the search engine: experts who participate by asking or answering questions with different perspectives and a moderator, a non-expert who knows enough to ask good questions and steers the discourse. the user can jump in at any time to steer the discourse and inject questions and opinions according to their interest. co-storm maintains a dynamic, hierarchical mind map to ensure they can easily follow and engage @xcite . upon the conclusion of the discourse, users can request the system to generate a cited report based on the mind map. for evaluation, we introduce wildseek, a dataset of topics and user goals from real users engaged in complex information-seeking across multiple domains. we propose automatic metrics to assess both discourse trace and final report quality. our results show that co-storm significantly outperforms rag chatbots in surfacing in-depth and serendipitous information while providing a more engaging learning experience. we further conduct a human evaluation by inviting 20 users with diverse backgrounds to compare co-storm with a search engine and a rag chatbot. 70% preferred co-storm over the search engine, and 78% preferred it over the rag chatbot for the overall information-seeking experience. participants find that co-storm facilitates serendipitous discovery and requires less mental effort. our main contributions include: ## co-storm implementation the lm component of co-storm is implemented using zero-shot prompting via the dspy framework @xcite and gpt-4o-2024-05-13 (see full prompts in appendix d). we ground co-storm on the internet using the you.com search api 5 although the system is compatible with other search engines or ir systems. hyperparameters n , k, l, α are set to 3, 10, 2, and 0.5, respectively. the text embeddings in eq. 1 are obtained from text-embedding-3-small. we set the lm temperature as 1.0 and top_p as 0.9 for all experiments. for human evaluation, we develop a web application (figure 12 ) for users to interact with co-storm in real time. ## automatic evaluation automatic evaluation enables scalable testing and allows for consistent simulation of user behavior. we compare co-storm with the following baselines: (1) rag chatbot, a baseline that retrieves information from the search engine and interacts with the user through a one-question-one-answer paradigm. (2) storm + qa, a baseline that uses the storm framework @xcite to generate a report for a given topic to provide general information. it then allows the user to ask followup questions and provides corresponding answers retrieved with the search engine. ## human evaluation human evaluation is essential for assessing systems designed for collaborative discourse, as it captures the complexities of human interaction, reflects familiar real-world interactions, and provides critical insights into the system's effectiveness. we compare co-storm with two baselines: (1) rag chatbot as detailed in section §5. (2) traditional search engine. ## conclusion we propose co-storm, an information-seeking assistance system that emulates collaborative discourse among users and multiple lm agents. by creating an interactive environment where users can both observe and participate, co-storm enhances learning and the complex information-seeking process. to facilitate automatic evaluation, we construct the wildseek dataset, which captures the information-seeking needs and intents of real internet users. experimental results, including extensive human assessments, show that co-storm outperforms traditional search engines and rag chatbots in surfacing unknown unknowns for human learning and reducing users' mental effort.