W eb CPM : Interactive Web Search for C hinese Long-form Question Answering (2023)

Abstract: Long-form question answering (LFQA) aims at answering complex, open-ended questions with detailed, paragraph-length responses. The de facto paradigm of LFQA necessitates two procedures: information retrieval, which searches for relevant supporting facts, and information synthesis, which integrates these facts into a coherent answer. In this paper, we introduce WebCPM, the first Chinese LFQA dataset. One unique feature of WebCPM is that its information retrieval is based on interactive web search, which engages with a search engine in real time. Following WebGPT, we develop a web search interface. We recruit annotators to search for relevant information using our interface and then answer questions. Meanwhile, the web search behaviors of our annotators would be recorded. In total, we collect 5,500 high-quality question-answer pairs, together with 15,372 supporting facts and 125,954 web search actions. We fine-tune pre-trained language models to imitate human behaviors for web search and to generate answers based on the collected facts. Our LFQA pipeline, built on these fine-tuned models, generates answers that are no worse than human-written ones in 32.5% and 47.5% of the cases on our dataset and DuReader, respectively. The interface, dataset, and codes are publicly available at https://github.com/thunlp/WebCPM .

## introduction long-form question answering (lfqa) @xcite targets answering complex, open-ended questions with detailed, paragraph-length responses. current lfqa solutions generally follow the retrieve-then-synthesize paradigm, which comprises two core ingredients: information retrieval and information synthesis. the former searches external knowledge sources (e.g., the web) for diverse * corresponding author. one defect of the conventional lfqa paradigm is that it often resorts to non-interactive retrieval methods, which use the original question as the query to retrieve a pile of uncurated information. on the contrary, humans are able to perform interactive web search by engaging with a search engine in real time. for a complex question, humans tend to decompose it into multiple sub-questions and ask them in sequence. by identifying and browsing relevant information, humans can improve their understanding of the topic and refine their searches by asking follow-up questions or related terms. this iterative process enables expanding the scope of their searches and improving the results they receive. overall, interactive web search not only provides access to diverse information sources, but also reflects the cognitive process of how humans solve questions, which allows for better interpretability. webgpt @xcite is one pioneering work that supports interactive web search for lfqa. the authors first build a web search interface backed up by microsoft bing, then recruit annotators to collect information using the interface to answer questions. after that, they fine-tune @xcite to imitate human behaviors for web search and to organize the collected information into answers. in the experiments, we-bgpt shows exceptional ability in lfqa, even surpassing human experts. despite its impressive performance, webgpt still remains mysterious to the community. this is because webgpt's interface, dataset, and trained models are not publicly available, and the inner workings of its core design elements remain opaque. these factors make it hard for the community to understand the challenges of interactive web search for lfqa and to continue exploring this line of study. resource webcpm (this work) dureader @xcite cmrc @xcite c 3 @xcite webgpt @xcite gophercite @xcite language in view of this, we deem it urgent to provide an accessible platform and public benchmark for this area. to this end, we first construct an interface (figure 1 ) to record web search behaviors when humans gather relevant information for longform questions. in the interface, users can execute pre-defined actions to perform multiple rounds of searching and browsing. when finding relevant information on a web page, they can record it as a supporting fact. meanwhile, their web-browsing behaviors will be recorded. after collecting enough information, users can finish the web search and answer the questions based on their collected facts. based on the interface, we choose chinese as the testbed and construct webcpm, focusing on interactive web search with chinese pre-trained models. webcpm is the first public qa dataset that involves interactive web search, and also the first dataset that targets chinese lfqa. webcpm contains 5, 500 question-answer pairs, together with 15, 372 supporting facts and 125, 954 web search actions. then we propose a general framework consisting of (1) a search model, which imitates human web search behaviors for information retrieval. specifically, the search model comprises three modules to execute a series of pre-defined actions on our interface: an action prediction module, a search query generation module, and a supporting fact extraction module; (2) a synthesis model, which generates a coherent answer conditioned on the collected facts. in the experiments, we choose 8 representative pre-trained language models (plms) with up to 10b parameter size, and evaluate their ability of interactive web search and information synthesis. we find that scaling model sizes is critical to achieving better performance. by selecting the best-performing backbone plm for the search and synthesis model, we combine them into a holistic lfqa pipeline and compare its capability with humans. human evaluation reveals that our pipeline generates answers that are no worse than humans 32.5% of the time on our test set. when applied to questions whose annotated answers are longer than 400 chinese characters from dureader @xcite , our pipeline generates answers that are better than golden annotated ones 47.5% of the cases, showing satisfying out-of-distribution generalization performance. we also show that our search model surpasses the conventional noninteractive retrieval method. finally, we analyze the contribution of core design elements of our framework and the human-like behaviors our models acquire. we envision these resources to serve as the testbed for other research topics, such as behavior cloning @xcite and tool learning @xcite . ## web search environment following webgpt, we construct a text-only interface to record web search behaviors when humans gather relevant information for long-form questions. our interface, backed up by bing search api, supports 10 mainstream web search actions as shown in figure 1 . when an action is executed, our interface responds with changes in the window. when the action search is performed, the interface enters search mode (figure 1 ), which displays the links recommended by bing for a specific @xmath0. each link comprises a title and a brief snapshot of the specific web page. each window displays three links one time, and more links can be accessed by executing the scroll down action. when finding the i-th link in the current window to be relevant, users could execute the load @xmath1, 2, 3}). the interface would enter the browsing mode (figure 6 in the appendix) and render the texts cleaned from the html of @xmath2. the content users could view at a time in the window is restricted up to 500 chinese characters, and more content can be accessed with the scroll action. users can utilize the quote action to extract consecutive sentences in the current window as a supporting fact. to enable extracting texts that stretch across two windows, the merge action is designed to merge the last two facts into a single fact (see appendix a.2 for more details). we also display all the existing extracted supporting facts for users. after browsing the i-th page, users can return to the previous search mode using the go back action to access other links. meanwhile, a refined query can be sent at any time. in general, users can freely interact with our interface multiple times until executing the finish action or triggering the maximum number of actions (100 in our case). the interface would automatically record meaningful actions and observations during web search. owing to the multilingual nature of bing system, although this work focuses on chinese, our interface can be flexibly adapted to other languages as well. for more technical details, please refer to appendix a. ## data collection we employ 23 annotators from different walks of life, who are experienced in search engine operation. we ask them to answer long-form questions by first searching for relevant information using our interface, then writing a nuanced answer. for quality control, we recruit 8 experts familiar with qa research as quality inspectors. next, we introduce the construction process of our dataset, with detailed annotation guides left in appendix b. question creation. creating new long-form questions from scratch without any reference is counterproductive, thus we turn to public qa forums as the question source. specifically, we engage annotators to refer to the questions on an english qa forum reddit, and then create new questions written in chinese. the details of this creation process are elaborated in appendix c. we find empirically that questions created in this way often necessitate multiple rounds of searching and browsing to collect sufficient information. interactive web search. given a question, we ask annotators to search for accurate and relevant information from trusted sources using our interface. this process may involve sending refined queries to bing multiple times, as well as exploring various web pages they deem to be relevant. we require annotators to carefully judge the factual accuracy of the information before extracting it as a supporting fact. the search process would be finished until sufficient supporting facts are collected. among our created questions, 26.2% are unanswerable and finally discarded because annotators cannot find sufficient useful information. answer annotation. after gathering enough supporting facts, the annotators would write selfcontained answers based on their collected information. we give them instructions for answer annotation, including writing answers that are relevant to the question and have rich content, maintaining logical consistency, clarity, and coherence, and providing viewpoints in an unbiased manner. quality control. each annotated instance is checked and approved by the quality inspectors before being selected for the final dataset. first, inspectors would manually inspect the action sequences recorded on the interface and discard lowquality ones (e.g., those with evident clerical errors in the issued queries). second, they would carefully check the collected supporting facts. if these facts are apparently insufficient to answer the question, irrelevant to the question, or factually incorrect, the corresponding action sequence would be abandoned. the above procedures remove 25% collected instances. for the remaining instances, inspectors would carefully examine their annotated answers. if an answer contradicts the abovementioned instructions, inspectors would return it to annotators and point out which requirement is not satisfied. annotators would revise their answers possibly for multiple rounds until the revised answer is up to standard. dataset statistics. ultimately, we collect 5, 500 instances, each formatted in a tuple of (question, web search behavior, supporting fact, answer), and also record the observations at each action execution. we display an example in figure 2 for reference, where we present the following: the original question, the simplified action sequence, the collected supporting facts, and the annotated answer. we partition the dataset into {4, 700, 400, 400} as the training, development, and test set. on average, each question involves performing 22.9 actions, sending 2.5 queries, and loading 3.3 web pages. the detailed proportion of each action is visualized in figure 7 in the appendix. ## framework in this section, we introduce how to teach plms for (1) interactive web search using our interface ( § 5.1) and (2) information synthesis ( § 5.2). the overall framework is illustrated in figure 3 . ## experiments and analyses our problem consists of 4 sub-tasks: action prediction, search query generation, supporting fact extraction, and information synthesis. correspondingly, we first train 4 modules and evaluate each sub-task independently by feeding the ground truth input to each module ( § 6.1). then we combine all modules into a unitary pipeline and only feed the question to the pipeline for a holistic evaluation ( § 6.2). finally, we conduct in-depth analyses for each module to understand their behaviors ( § 6.3). ## conclusion in this paper, we construct a benchmark of interactive web search for chinese long-form qa, together with an open-source interface. we decompose the task into 4 sub-tasks and design a modular pipeline. by fine-tuning representative plms, we conduct both an individual evaluation for each module and a holistic evaluation for the pipeline. in-depth analyses are carried out to understand the core design elements of our framework. we expect our interface, dataset, framework, and analyses to facilitate more future explorations in this area.