Navigating the Path of Writing: Outline-guided Text Generation with Large Language Models (2025)

Abstract: Large Language Models (LLMs) have impacted the writing process, enhancing productivity by collaborating with humans in content creation platforms. However, generating high-quality, user-aligned text to satisfy real-world content creation needs remains challenging. We propose WritingPath, a framework that uses explicit outlines to guide LLMs in generating goal-oriented, high-quality text. Our approach draws inspiration from structured writing planning and reasoning paths, focusing on reflecting user intentions throughout the writing process. To validate our approach in real-world scenarios, we construct a diverse dataset from unstructured blog posts to benchmark writing performance and introduce a comprehensive evaluation framework assessing the quality of outlines and generated texts. Our evaluations with various LLMs demonstrate that the WritingPath approach significantly enhances text quality according to evaluations by both LLMs and professional writers.

## introduction writing is a fundamental means of structuring thoughts and conveying knowledge and personal opinions @xcite . this process requires systematic planning and detailed review. @xcite describes writing as a complex problem-solving process and explores how planning and execution interact in writing. that is, writing involves more than merely generating text; it encompasses developing a proper understanding of the topic, gathering relevant subject matter, and implementing thorough structuring. recent advancements in large language models (llms) have advanced the writing workflow, enhancing both its efficiency and productivity. one significant area of exploration is the collaborative † work done as a research intern at naver figure 1 : comparative overview of writing approaches: (a) direct generation, (b) iterative writing involving planning, editing, and explaining, and (c) writingpath method, which starts with a consistency-focused plan, incorporates information-rich browsing, and results in an augmented, consistent, and rich outline. to address this, we propose writingpath, a methodology designed to incorporate user intentions such as desired topic, textual flow, keyword inclusion, and search result integration into the writing process. writingpath emphasizes the importance of systematic planning and a clear outline from the early stages of writing. inspired by the structured writing plan of @xcite and the reasoning path of @xcite , the writingpath collects ideas and creates outlines that encapsu-late the user's intentions before generating the final text. furthermore, the initial outlines are further augmented with additional information through information browsing. such a structured approach offers enhanced control over the text generation process and improves the quality of the content produced by llms. we also utilize a multi-aspect writing evaluation framework to assess the intermediate and final productions from the writingpath, offering a way to evaluate the quality of free-form text foot_0 without relying on reference texts. taking into account that conventional likert scales (1-5 ratings) (clark; hinkin, 1998) make it challenging to systematically compare and evaluate diverse writing outputs, particularly in creative tasks @xcite , our evaluation framework aims to provide more precise and reliable assessments for the outlines and final texts. for evaluation purposes, we construct a free-form blog text dataset incorporating a wide range of writing styles and topics from real users, including beauty, travel, gardening, cooking, and it. using this dataset, we evaluate how well the llm outputs reflect the user's intentions. applying the writingpath to various llms shows significant performance gains across all evaluated models. these results validate that our approach enables the models to maintain a stronger focus on the given topic and purpose, ultimately generating higher-quality text that more accurately reflects user intentions. furthermore, to validate real-world applicability, we applied writingpath to a commercial writing platform for beta testing from october 2023 to march 2024. the deployment demonstrated its effectiveness in supporting real users with structured content creation across diverse writing needs. the main contributions of this study can be summarized as follows: ## design of writingpath we propose writingpath, a systematic writing process to produce consistent, rich, and well-organized text with llms. inspired by human writing processes, it consists of five key steps: metadata preparation, initial outline generation, information browsing, augmented outline creation, and final text writing (figure 2 ). each step is guided by a specific prompt configuration that aligns llm output with specific step requirements. the core components of writingpath are those that generate outlines as they establish a structured writing plan. research suggests that a wellstructured outline significantly impacts the quality of the written text @xcite @xcite . the initial sketch is transformed into a detailed outline, including the flow, style, keywords, and relevant information from search results. this outline provides a clearer view of the final text to the llms. the specific steps are described as follows: step 1: prepare meta data the first step establishes the writing direction and target reader using metadata m, which includes i) purpose, ii) type, iii) style, and iv) keywords. to simulate this process, we converted human-written texts into metadata (see section 4.1 for details of the dataset). step 2: generate title and initial outline the second step generates the title t and initial outline o init based on the metadata m from step 1, using the llm function f llm with a prompt configuration function ϕ s . here, s indicates the step index, and for step 2, the prompt configuration is ϕ 2 : the initial outline o init consists of main headers h i,0 , where i denotes the header sequence. this outline serves as the scaffolding of the text, organizing the main ideas and laying out the key points. step 3: browse for information the third step enriches the text by collecting additional information and keywords to reinforce the initial outline. we use the search function f search with the generated title t as the query to retrieve the top-1 blog document, d sim : in our implementation, we employ the naver search api foot_1 to retrieve the top-1 document among similar blog posts. from the blog document, we extract keywords k using the f llm with a prompt configuration ϕ 3 : the extracted keywords constitute the additional information from the search results, leading to a more specific writing plan, improving the quality of the generated text. ## evaluation of writingpath evaluating the effectiveness of writingpath compared to existing writing support systems is challenging. most previous studies do not directly utilize outlines in the writing process, resulting in a lack of systematic methods to assess outline quality. even when outlines are used, evaluation relies only on human evaluation @xcite . moreover, current approaches heavily rely on human evaluation, which poses challenges for assessing full texts @xcite @xcite , as it requires evaluating multiple aspects of the written work. this challenge can arise in content creation workflows where scalable and consistent quality assessment helps maintain content standards. to address these limitations, we propose an evaluation framework that combines human and automatic evaluation to assess the quality of generated outlines and final texts from multiple perspectives. this hybrid approach is designed to support realworld content creation workflows by combining systematic automated metrics with human assessment of nuanced writing aspects that require subjective judgment. the proposed method establishes clear evaluation criteria, enabling objective and reproducible validation of writingpath's effectiveness as a writing support system. ## real-world deployment writingpath was integrated into a commercial blogging platform as a writing assistance feature and tested for six months. in the service environment, additional considerations such as safety filtering and content quality control measures were necessary for reliable content generation. the system architecture of clova for writing by naver is depicted in figure 7 . the serving pipeline integrates multiple components for reliable service operation. it integrates user request handling, content filtering, kafka pipeline, and retrieval. requests pass through a gateway with rate limiting and are filtered for harmlessness. specifically, the system includes emergency filtering and safety classification before passing requests to writingpath. additionally, a token event monitoring system tracks model usage, followed by token event filtering over output anomalies. ## conclusion we introduced writingpath, a framework that enhances the ability of llms to generate high-quality and goal-oriented writing by employing explicit outlines. designed for real-world content creation, our approach uses structured guidelines from the early stages to ensure consistent quality control. we verified the impact of writingpath by conducting a comprehensive evaluation that incorporates automatic and human evaluations covering a wide range of aspects. our experimental results demonstrate that texts generated following the full writingpath approach, which includes the use of augmented outlines, exhibit superior performance compared to texts produced using only initial outlines or without any intermediate outlines. we also proposed a framework for assessing the writing-path's intermediate outlines, which found that augmented outlines have better inherent quality than initial outlines, demonstrating the importance of outline augmentation steps. we hope that this work will contribute to the research and development of more reliable ai-assisted writing solutions.