Beyond Binary Preferences: Semi-Online Label-Free GRACE - KTO with Group-Wise Adaptive Calibration for High-Quality Long-Text Generation (2025)

Abstract: Generating high-quality long-text remains challenging for Large Language Models (LLMs), as conventional supervised fine-tuning fails to ensure overall quality due to its teacher-forcing nature. Kahneman-Tversky Optimization (KTO), as a model alignment method that can holistically optimize generation quality, overcomes the need for paired preference data required by previous methods. However, it still suffers from binary supervision that inadequately reflects varying quality degrees. To address this, we propose GRACE-KTO, a semi-online framework that transforms KTO’s binary signals into dynamically calibrated intra-group rewards. Specifically, GRACE-KTO aggregates responses to identical queries into groups, computes rank-sum scores across multiple linguistic quality dimensions, and applies group-wise and global normalization to adaptively redistribute sample importance. We adopt a semi-online training strategy to reduce costly online sampling while outperforming offline variants. By leveraging query generation with seed data, we minimize labeled data dependency, using the model’s own knowledge to enhance its long-text generation capabilities. Additionally, we extend the context window to 32k tokens using YaRN during inference, enabling the model to generate longer texts while maintaining perplexities. Experiments demonstrate GRACE-KTO’s superiority over vanilla KTO on both automatic metrics and LLM-as-a-Judge evaluations, advancing long-text generation through group-wise adaptive calibration.

## introduction ensuring high-quality long-text generation remains a formidable challenge for large language models (llms). while long-context llms have made remarkable progress in understanding lengthy texts with context lengths reaching 1m tokens or more @xcite , long-text generation presents a distinct and more complex task. this task demands that models produce content significantly longer than the input text. even powerful open-source models like qwen2.5-72b-instruct @xcite , despite their impressive capabilities, face limitations and can only generate up to 8k tokens. this disparity highlights the ongoing need for advancements in training methodologies to overcome the inherent challenges of long-text generation. conventional supervised fine-tuning (sft) for long-text generation relies on teacher-forcing @xcite to imitate training sequences stepwise, but has limited capacity to enhance holistic text quality. this arises from exposure bias @xcite : models trained on ground-truth contexts are incapable of handling errors that accumulate during autoregressive generation. in long-text generation, early inaccuracies trigger cascading errors through subsequent to-kens, worsening the training-inference gap between teacher-forced optimization and free-generation execution. this gap ultimately restricts sft's ability to improve the overall quality of the responses. to address this limitation, various alignment methods have been proposed to optimize the generation quality from a more comprehensive perspective. among them, kahneman-tversky optimization @xcite (kto) has shown promising results. unlike previous methods like proximal policy optimization @xcite (ppo) and direct preference optimization @xcite (dpo) that require paired preference data, kto can work with unpaired data, offering greater flexibility. kto is based on kahneman-tversky's prospect theory @xcite , which models human utility in a way that captures biases such as loss aversion. however, kto still employs binary supervision signals, which are insufficient to capture the varying degrees of quality among different generated texts. this limitation hinders kto's ability to fully utilize the available data and accurately guide the model towards generating higher-quality long-text. in this work, we propose group-reward adaptive calibration for enhanced kto (grace-kto), a semi-online framework aimed at enhancing long-text generation quality by addressing the limitations of kto. grace-kto refines kto's binary signals into dynamic intra-group rewards by harnessing kto's flexibility with unpaired data. the process begins by grouping responses to identical queries and calculating rank-sum scores across key linguistic dimensions, including content length, expression diversity, professionalism, and relevance to the query. these scores are then normalized within each group to provide a comprehensive reflection of each response's quality. moreover, global normalization is applied across all samples to adaptively adjust the importance of each sample, thereby determining their respective weights in the kto training process. to improve both efficiency and effectiveness, we adopt a semi-online training strategy. unlike fully online algorithms that require frequent realtime model updates for sampling small batches of data-a time-consuming process particularly for large models like 72b models-our semi-online approach efficiently harnesses the concurrent processing power of frameworks like vllm @xcite to significantly enhance sampling ef-ficiency. furthermore, this strategy proves to be more effective than offline training methods. to reduce the reliance on labeled data, we utilize the training set as seed data and prompt the llm to generate new queries for training. this allows the model to improve the quality of its responses by leveraging its inherent knowledge rather than depending solely on labeled data. additionally, by extending the context window to 32k tokens using yarn @xcite interpolation during inference, the model can generate longer texts with low perplexities. through experiments, we show that grace-kto surpasses vanilla kto in both automatic metrics and llm-as-a-judge @xcite evaluations. by dynamically calibrating rewards across groups, grace-kto enables the model to better learn from varying quality degrees in generated texts, resulting in more coherent and contextually consistent long-text outputs. in summary, our work makes the following key contributions: ## methodology in this section, we introduce grace-kto, our proposed framework for enhancing the quality of long-text generation. as depicted in figure 2 , grace-kto refines the conventional binary feedback of kto through group-wise adaptive calibration. this process enriches the training information by capturing diverse quality degrees. our semi-online training approach not only improves sampling efficiency but also outperforms purely offline training. by using the training set as seed data and employing query generation to construct an alignment training dataset, grace-kto leverages the model's own knowledge to enhance its performance. furthermore, by extending the context window, we enable the generation of longer texts while maintaining low perplexity. ## conclusion in this paper, we introduce grace-kto, a semionline framework designed to enhance the quality of long-text generation. grace-kto aggregates responses into groups and calculates ranksum scores across multiple linguistic quality dimensions. this allows it to adjust intra-group rewards and adaptively redistribute sample importance through group-wise and global normalization, fully utilizing kto's flexibility with unpaired data. we implement a semi-online training strategy to minimize expensive online sampling costs and reduce reliance on labeled data by generating queries from seed data. additionally, we extend the context window to 32k tokens using yarn during inference. experiments show grace-kto outperforms vanilla kto. overall, our work provides a new and effective approach to long-context generation. future research will focus on enhancing the reward strategy of grace-kto by integrating more structured evaluation signals, and applying it to multi-domain text generation.