L ong DPO : Unlock Better Long-form Generation Abilities for LLM s via Critique-augmented Stepwise Information (2025)

Abstract: Recent advancements in large language models (LLMs) have markedly improved their capacity to handle long text inputs; however, current models, including GPT-4o, still exhibit unsatisfactory performance in long-form generation. Generating high-quality long-form content still remains a significant challenge. In this paper, we present LongDPO, a novel approach designed to enhance long-form text generation through step-level supervision. By leveraging Monte Carlo Tree Search (MCTS) to collect stepwise preference pairs and employing a global memory pool to maintain factual accuracy, LongDPO effectively mitigates issues such as inconsistencies that are prevalent in long-context LLMs. Furthermore, we integrate critique-augmented generation to refine the selected preference pairs. Following the collection of stepwise preference pairs, we apply stepwise preference learning for fine-grained optimization. Experimental results demonstrate that our method enhances performance on long-form generation benchmarks (e.g. LongBench-Write) while maintaining nearly lossless performance on several general benchmarks.

## introduction recent advancements in large language models (llms) @xcite @xcite , have significantly enhanced their capacity to process long text sequences with models like gpt-4o now capable of handling contexts up to 128k tokens @xcite . despite these strides, there has been less emphasis on the models' ability to generate better long-form text outputs. the capability to produce long-form content is essential for various real-world applications, including writing academic papers, novels, and scripts in literature, generating legal contracts in law, and producing repository-level code in technology @xcite . however, many llms still struggle to generate content exceeding 2,000 words @xcite , highlighting the need for further advancements in this area. previous research has explored methods to extend the output window by creating long-form training data and leveraging preference learning. for example, suri @xcite creates various instructions for the same response and performs outcome-level preference optimization. longwriter @xcite employs an agentbased pipeline that decomposes ultra-long generation tasks into subtasks to build a long-form dataset, followed by supervised fine-tuning and dpo. these approaches primarily rely on outcome supervision @xcite during dpo, which provides feedback on the final result, for long-form generation tasks. nevertheless, long-context llms are more prone to produce responses with issues such as logical inconsistencies, fabricated content, and failure to fully meet query requirements @xcite . these challenges make outcome supervision, which directly provides feedback for a long sequence, particularly problematic. in contrast, process supervision involves supervising each intermediate step, which offers more granular and precise feedback. furthermore, process supervision specifies the exact location of low-quality steps, thereby facilitating the refinement of these steps @xcite . consequently, breaking down a long sequence into intermediate steps and supervising these shorter steps could be a more effective strategy. in this paper, we introduce longdpo, which enhances long-form generation capabilities through step-level supervision. longdpo first constructs preference data with stepwise supervision and then performs stepwise learning. specifically, we use monte carlo tree search (mcts) @xcite to collect stepwise preference pairs. considering that long-context llms are prone to generating inconsistent content, leading hallucinations @xcite , we incorporate a global memory pool to improve the factual consistency of the selected preference pairs. additionally, the quality of candidates generated heavily relies on the original model's inherent capability. simply searching for candidates is both inefficient and ineffective @xcite . to address this, we propose critique-augmented generation to obtain better candidates for the selected preference pairs. after gathering the stepwise preference pairs, we propose employing a stepwise dpo for finegrained learning. as illustrated in figure 1 , traditional dpo applies sample-wise supervision directly, which can lead to a less pronounced reward margin, complicating the learning process @xcite . in contrast, longdpo utilizes finegrained learning at each step, which has the potential to produce superior results. we evaluate long-form generation capabilities using longbench-write-en and longgen-bench @xcite , which assess text generation length, quality, and adherence to instructions. additionally, we use general benchmarks such as truthfulqa @xcite to measure overall task performance. our method, built on llama-and qwen-based backbones, outperforms their vanilla dpo versions in long-form generation tasks while maintaining near-lossless performance on general tasks. our contributions can be summarized as follows: ## related work long context llms some studies explore to extend the input context window, using training-based methods like @xcite @xcite and training-free methods, such as @xcite @xcite . many llms can support input context windows of 128k. however, far fewer are capable of generating outputs exceeding 2k words in length. recent studies @xcite ) have employed outcome supervision to extend the output window. most recently, @xcite proposed longreward, which is orthogonal to our work. however, in addition to the instruction and response, it requires an additional reference long document as input, which limits its applicability in both outcome and process supervision. another line of exploration in long-text generation, such as hierarchical writing and recurrent prompting @xcite @xcite , is orthogonal to our method. process supervision in preference learning recently, scaling inference-time compute has become increasingly popular @xcite @xcite . process supervision with mcts can further enhance models' reasoning abilities @xcite . recent studies @xcite use mcts in both math and code tasks. in addition to mcts, @xcite also incorporate self-reflection. cheng et al. (2024) employ tree search and train a refiner for iterative optimization. in this work, we primarily focus on exploring prepare data and generate critiques critique-augmented generation input principle selected ＋ … … … 1 chosen rejected refined 2 3 3 consistent sibling nodes powerful model factual content ## longdpo our method consists of two main parts: 1) collecting stepwise preference data, and 2) using the collected preference data for dpo training. ## disscussion longdpo focuses on long-form tasks (e.g., creative writing), which, unlike tasks such as math and coding, do not have a ground truth. it is more challenging to assess the reward precisely. different from existing literature in reinforcement learning, which can rely on rule-based rewards or process reward models, we take into full consideration the characteristics of natural language and have carefully designed seven principles for evaluating the reward. ## conclusion in this paper, we propose longdpo which incorporate process supervision with mcts to collect better preference pairs with a memory pool to maintain factual consistency and leverages external critiques to refine low-quality candidates in long-form generation. longdpo enhances performance in long-form generation tasks (e.g. longbench-write) while maintaining near-lossless performance on several general tasks.