Assisting in Writing W ikipedia-like Articles From Scratch with Large Language Models (2024)

Abstract: We study how to apply large language models to write grounded and organized long-form articles from scratch, with comparable breadth and depth to Wikipedia pages. This underexplored problem poses new challenges at the pre-writing stage, including how to research the topic and prepare an outline prior to writing. We propose STORM, a writing system for the Synthesis of Topic Outlines throughRetrieval and Multi-perspective Question Asking. STORM models the pre-writing stage by (1) discovering diverse perspectives in researching the given topic, (2) simulating conversations where writers carrying different perspectives pose questions to a topic expert grounded on trusted Internet sources, (3) curating the collected information to create an outline.For evaluation, we curate FreshWiki, a dataset of recent high-quality Wikipedia articles, and formulate outline assessments to evaluate the pre-writing stage. We further gather feedback from experienced Wikipedia editors. Compared to articles generated by an outline-driven retrieval-augmented baseline, more of STORM’s articles are deemed to be organized (by a 25% absolute increase) and broad in coverage (by 10%). The expert feedback also helps identify new challenges for generating grounded long articles, such as source bias transfer and over-association of unrelated facts.

## introduction large language models (llms) have demonstrated impressive writing capabilities @xcite @xcite , but it is unclear how we can use them to write grounded, long-form articles, like full-length wikipedia pages. such expository writing, which seeks to inform the reader on a topic in an organized manner @xcite , requires thorough research and planning in the pre-writing stage (rohman, figure 1 : we explore writing wikipedia-like articles from scratch, which demands a pre-writing stage before producing the article. in this stage, simpler approaches like direct prompting have limited planning capacity. in contrast, storm researches the topic via perspectiveguided question asking in simulated conversations. we explore these challenges by focusing on how to generate wikipedia-like articles from scratch. we decompose this problem into two tasks. the first is to conduct research to generate an outline, i.e., a list of multi-level sections, and collect a set of reference documents. the second uses the outline and the references to produce the full-length article. such a task decomposition mirrors the human writing process which usually includes phases of pre-writing, drafting, and revising @xcite . as pre-trained language models inherently possess a wealth of knowledge, a direct approach is to rely on their parametric knowledge for generating outlines or even entire articles (direct gen). however, this approach is limited by a lack of details and hallucinations @xcite , particularly in addressing long-tail topics @xcite . this underscores the importance of leveraging external sources, and current strategies often involve retrieval-augmented generation (rag), which circles back to the problem of researching the topic in the pre-writing stage, as much information cannot be surfaced through simple topic searches. human learning theories @xcite highlight asking effective questions in information acquisition. although instruction-tuned models @xcite can be prompted directly to generate questions, we find that they typically produce basic "what", "when", and "where" questions (figure 1 the design of storm is based on two hypotheses: @xcite diverse perspectives lead to varied questions; (2) formulating in-depth questions requires iterative research. building upon these hypotheses, storm employs a novel multi-stage approach. it first discovers diverse perspectives by retrieving and analyzing wikipedia articles from similar topics and then personifies the llm with specific perspectives for question asking (figure 1 (b) ). next, to elicit follow-up questions for iterative research (figure 1 (c)), storm simulates multi-turn conversations where the answers to the generated questions are grounded on the internet. finally, based on the llm's internal knowledge and the collected information, storm creates an outline that can be expanded section by section to develop a fulllength wikipedia-like article. we evaluate storm using our freshwiki dataset ( §2.1) which curates recent, high-quality wikipedia articles to avoid data leakage during pretraining. 1 to facilitate the study of the pre-writing stage, we define metrics for evaluating the outline quality against human-written articles. we further invited a group of experienced wikipedia editors for expert evaluation. the editors found storm outperforms an outline-driven rag baseline, especially regarding the breadth and organization of the articles. they also identified challenges for future research, including addressing cases where: (1) the bias on the internet affects the generated articles; (2) llms fabricate connections between unrelated facts. these challenges present new frontiers to grounded writing systems. our main contributions include: ## freshwiki we study generating wikipedia-like articles from scratch, placing emphasis on the pre-writing stage @xcite , which involves the demanding sub-tasks of gathering and curating relevant information ("research"). this models the human writing approach which has prompted some educators to view wikipedia article writing as an educational exercise for academic training @xcite . our setup emphasizes the capability of longform grounded writing systems to research and curate content. specifically, given a topic t, the task is to find a set of references r and generate a full-length article @xmath0...s n , where each sentence s i cites a list of documents in r. foot_1 ## method we present storm to automate the pre-writing stage by researching a given topic via effective question asking ( §3.1, §3.2) and creating an outline ( §3.3). the outline will be extended to a fulllength article grounded on the collected references ( §3.4). figure 2 gives an overview of storm and we include the pseudo code in appendix b. ## human evaluation to better understand the strengths and weaknesses of storm, we conduct human evaluation by collaborating with 10 experienced wikipedia editors as a general framework, rag is flexible in both the retrieval source and time. the retrieval sources can vary from domain databases @xcite , code documentation @xcite , to the whole internet @xcite . regarding the time, besides a onetime retrieval before generation, the system can be designed to self-decide when to retrieve across the course of the generation @xcite @xcite . automatic expository writing different from other types of long-form generation @xcite , automatic expository writing requires grounding on external documents and leveraging the interplay between reading and writing. @xcite propose the imitate-retrieve-paraphrase framework for expository writing at the paragraph level to address the challenges in synthesizing information from multiple sources. beyond summarizing sources, @xcite highlight that expository writing requires the author's sensemaking process over source documents and good outline planning. we tackle these challenges by focusing on the pre-writing stage. question asking in nlp question asking capabilities in nlp systems have expanded across several fronts, including generating clarification questions to understand user intents @xcite , and breaking large questions into smaller ones to improve compositional reasoning @xcite . while humans usually ask questions to learn new knowledge @xcite , how to optimize question informativeness and specificity in information-seeking conversations remains less explored. the closest work is @xcite which defines the question informativeness using the unigram precision function and uses reinforcement learning to increase the question informativeness. ## conclusion we propose storm, an llm-based writing system that automates the pre-writing stage for creating wikipedia-like articles from scratch. we curate the freshwiki dataset and establish evaluation criteria to study the generation of grounded longform articles. experimental results demonstrate that the question asking mechanism in storm improves both the outline and article quality. with the improved breadth and depth, storm helps surface new challenges for grounded writing systems through expert evaluation. the experienced wikipedia editors in our study unanimously agree that storm is helpful for their pre-writing stage.