Rationalize and Align: Enhancing Writing Assistance with Rationale via Self-Training for Improved Alignment (2025)

Abstract: A Writing Assistant (WA) is a system that offers writing suggestions based on user instructions. Existing WAs are typically built by training large language models (LLMs) on domain-specific instruction data through supervised fine-tuning (SFT) only. However, SFT optimizes models to match a single reference, failing to capture the inherent flexibility of text editing, where multiple valid revisions exist. Therefore, solely relying on SFT limits WA performance. To address this limitation, we propose the Rationalize and Align framework, which enhances the WA performance with rationale (i.e., linguistic explanations) and alignment. Our framework automatically generates the rationale and preference data for writing tasks via distillation and self-training, eliminating the need for human annotation. These data are then leveraged to refine WA using a novel preference optimization method. Empirical results show that our framework significantly improves WA performance. Our WA outperforms both open-source state-of-the-art WAs and the closed-source GPT-4o by 3.9 and 7.1 points on average, respectively, across eight well-established writing-related test sets.

## introduction large language models (llms) demonstrate strong proficiency in generating fluent and coherent text revisions, assisting users with various writing tasks @xcite . however, general-purpose llms, such as gpt-4, often underperform compared to task-specific models @xcite . consequently, research has focused on tailoring llms for writing-related tasks, such as grammatical error correction (gec) @xcite ), text simplification @xcite , and style transfer @xcite . several studies @xcite have proposed writing assistants (was) that 1 https://github.com/nusnlp/rationalign ## rationalize & align was aim to generate a proper writing suggestion s based on the user's instruction and the input text. existing was @xcite focus solely on providing writing suggestions, which are trained to learn the mapping from x to s, where x represents the combination of instruction and input text. here, we want to enhance the wa by providing a rationale (or explanation) e for its s. therefore, our goal is to learn a mapping from x to y, where y represents the combination of suggestion s and rationale e. fig. 2 shows the method to build our wa, comprising three key components: rationale generation, reward modeling, and preference optimization. ## conclusion in this work, we introduce a novel framework, rationalize & align, to enhance the performance of llms in writing-related tasks. its core idea lies in distilling rationales from gpt-4o into the wa system and further improving it through self-training alignment, which includes reward modeling and preference optimization. our experiments demonstrate that each component contributes to superior writing suggestions. our wa surpasses both the sota wa system and gpt-4o in suggestion quality. moreover, we built the first open-source wa capable of generating rationales for its suggestions.