Token-level Preference Self-Alignment Optimization for Multi-style Outline Controllable Generation (2025)

Abstract: Multi-style outline controllable generation is crucial for multiple applications, including document semantic structuring and retrieval-augmented generation.The great success of preference alignment approaches encourages their application in controllable generation tasks.However, these attempts encounter several limitations: (1) response pair requirements, (2) substantial computation costs, and (3) insufficient exploitation of fine-grained preference signals.To address these problems, we propose a token-level preference self-alignment optimization, named TKPO, for outline controllable generation. TKPO extends the Bradley-Terry model from pair-wise to list-wise comparison, which is further applied at the token level for fine-grained preference signal utilization. In comparison to the representative methods, e.g., DPO, TKPO does not require response pairs; instead, we propose a controllable attributes-driven method to construct reject samples for self-alignment. Additionally, TKPO optimizes only the base model, thereby avoiding additional memory usage and substantial computational costs.We curate two outline controllable generation datasets with regard to language style and level-of-detail.Extensive experiments demonstrate that TKPO outperforms DPO by up to 19.28% in performance while requiring only 56.25% in training time.We release the code and datasets resources at https://github.com/WHUIR/TKPO.

## introduction given an article, outline controllable generation aims to output a hierarchical outline with respect to the content of the articles and the controllable conditions, as shown in figure 1 . existing attempts to outline generation can be classified into two research lines: (1) segmentation and summarization (sun et al., 2022); (2) prompt-driven generation (yang et al., @xcite . the first category divides articles into distinct parts based on paragraph coherence or topical consistency and then generates a summary or topic for each segment. prompt-driven approaches focus on tailored prompts fine-tuning for outline generation. nevertheless, it is noteworthy that the exploration of this task is insufficient, and none of the research focuses on the multi-style outline controllable generation, which is significant for various downstream applications, such as personalized article summarization, semantic structuralization, and retrieval augmented generation. to fill this gap, we first define two distinct outline styles, i.e., language styles (literary vs. objective) 1 and level-of-detail (concise vs. moderate vs. comprehensive) 2 (ref. inspired by the promising results of preference alignment algorithms in conditional generation @xcite , we are committed to applying these methods for outline controllable generation. however, existing preference alignment solutions encounter three limitations: (1) response pair requirements, (2) extensive computation costs, and (3) inadequate utilization of fine-grained preference signals, hindering their further development. more concretely, the majority of strategies necessitate a response pair (an accepted and a declined response) for preference comparison and alignment. this requirement poses significant challenges in the real world compared to the simpler single-response dataset construction. besides, these methods, such as dpo @xcite , involve at least two models (i.e., reference model and policy model) during training, incurring additional memory usage and computational costs. furthermore, we argue that sentence-level alignment limits the llms' discrimination capability @xcite , as the preferences are more susceptible to the specific words (e.g., toxic terms) @xcite . hence, fine-grained token-level signals distinguishing between good and bad responses are more appropriate for precise training and optimization. to address these issues, we propose a tokenlevel preference self-alignment optimization, dubbed tkpo. compared to existing token-level preference alignment attempts @xcite , our tkpo is response pair-free and efficient, enabling precisely controllable generation. specifically, tkpo expands the bradley-terry model @xcite from the pair-wise into the list-wise comparison (i.e., one accepted instance vs. multi-rejected instances), which is further applied at the token level. tkpo eliminates the necessity for paired response data, instead, we propose a controllable attribute-driven method, in which the fundamental capability of llms is exploited to construct reject samples for self-alignment. in comparison to the representative preference alignment methods, tkpo only optimizes the base model, avoiding additional memory usage and substantial computational cost. theoretical analysis demonstrates that optimizing the tkpo is equivalent to optimizing the upper bound of the optimal policy model under the bradley-terry framework. additionally, our tkpo demonstrates superior capability for optimizing the hard tokens for fine-grained alignments and exhibits notable diversity in the generation. we devise two metrics for the outline language style and level-of-detail controllable generation evaluation. extensive experiments illustrate that our method outperforms dpo with only 73.55% memory usage. to encapsulate, the main contributions of this paper include the following: ## conclusion we introduce a multi-style outline controllable generation task and curate two datasets regarding the outline language style and level-of-detail. additionally, a token-level preference self-alignment method, named tkpo, is proposed for a more efficient and precise controllable generation. specifically, tkpo extends the bradley-terry model from pair-wise to list-wise comparison and further applies it to token-level alignment. tkpo does not require response-pair for model training, instead, a controllable attributes-driven selfalignment method is applied. mathematical analysis demonstrates that tkpo confers benefits in output diversity and hard token optimization. besides, optimizing tkpo is tantamount to optimizing the upper bound of the optimal policy. compared to dpo, our tkpo achieves up to 19.28% performance improvement with only 73.55% memory usage and 56.25% training time.