GTA : Supervised-Guided Reinforcement Learning for Text Classification with Large Language Models (2025)

Abstract: In natural language processing (NLP) tasks, pure reinforcement learning fine-tuning methods often suffer from inefficient exploration and slow convergence; while supervised fine-tuning (SFT) methods, although efficient in training, have limited performance ceiling and less solid theoretical foundation compared to reinforcement learning. To address efficiency-capability trade-off, we propose the Guess-Think-Answer (GTA) framework that combines the efficiency of SFT with the capability gains of RL in a unified training paradigm. GTA works by having the model first produce a provisional guess (optimized via cross-entropy loss), then reflect on this guess before generating the final answer, with RL rewards shaping both the final output and the format of the entire GTA structure. This hybrid approach achieves both faster convergence than pure RL and higher performance ceiling than pure SFT. To mitigate gradient conflicts between the two training signals, we employ loss masking and gradient constraints. Empirical results on three text classification benchmarks demonstrate that GTA substantially accelerates convergence while outperforming both standalone SFT and RL baselines.

## related work cot prompting has emerged as a powerful technique to enhance the reasoning capabilities of llms by guiding them to generate intermediate reasoning steps before producing final answers @xcite @xcite . while cot significantly improves performance across various reasoning tasks, its application in text classification often requires extensive human annotation of reasoning chains, leading to substantial costs and quality inconsistencies @xcite . to address these limitations, researchers have explored rl approaches that can optimize model behavior without requiring manual annotation of intermediate steps @xcite . traditional rl methods in nlp, however, often suffer from inefficient exploration and slow convergence, making it challenging for pure rlbased methods to consistently outperform sft approaches despite their stronger theoretical foundation @xcite . recent advancements like the grpo algorithm @xcite have improved rl efficiency by estimating baselines from group scores, thereby reducing computational costs. additionally, when combining multiple learning objectives-such as sft and rl-gradient conflicts can arise, leading to suboptimal convergence. techniques such as gradient masking and cosine similarity adjustments have been developed to mitigate these conflicts by aligning gradients toward compatible directions @xcite , which inspires our approach to harmonizing supervised and rl signals within our proposed gta framework. ## methodology in this section, we present the proposed gta framework in detail. we begin with an overview of the overall design, followed by the description of each component and training objective, highlighting how the method effectively integrates reinforcement learning with reasoning-oriented supervision. ## experimental setup this section outlines the experimental setup used to evaluate our approach. we describe the datasets, baseline models, and implementation details. ## results and analysis in this section, we report the main experimental results and provide in-depth analyses. we first compare gta with baseline methods on multiple benchmarks, then investigate convergence behav-ior, reasoning robustness, and case studies to gain further insights into its effectiveness. ## conclusion and future work in this work, we present gta, a novel training framework that addresses the efficiency-capability trade-off between sft and rl by introducing a guess stage to traditional cot outputs. under this framework, model outputs are organized into guess, think, and answer segments, where the guess is optimized via sft while the overall format and final output are optimized through rl. to mitigate gradient conflicts, we apply loss masking and cosine-similarity constraints. experimental results on four text-classification benchmarks show that gta consistently outperforms pure sft and grpo baselines in accuracy and f 1 scores while achieving significantly faster convergence than pure rl approaches. through analysis of training dynamics, we demonstrate that gta successfully combines the efficiency of supervised learning with the performance gains of rl, substantially accelerating convergence and addressing the exploration inefficiency inherent to rl-based fine-tuning. theoretically, our approach is not limited to text classification tasks, and we plan to explore extending gta to broader nlp tasks in future work.