Curr- R e FT : Overcoming Training Bottlenecks in Small-scale Vision-Language Models via Curriculum Reinforcement Finetuning (2025)

Abstract: State-of-the-art vision-language models (VLMs) require massive scaling that limits practical deployment. Small-scale VLMs offer a practical alternative but face out-of-domain (OOD) collapse when trained with traditional supervised fine-tuning (SFT). Through GeneralPoints experiments, we identify that OOD collapse is due to SFT’s tendency to induce visual hallucinations under distribution shifts, whereas Reinforcement Learning’s (RL) bidirectional reward-driven mechanism with iterative error correction refines visual perception. Although RL-based post-training effectively mitigates OOD degradation, it faces a critical sparse reward dilemma in complex visual reasoning tasks. To this end, we propose Curriculum Reinforcement Finetuning (Curr-ReFT), comprising two sequential stages: (1) Structured Curriculum Reinforcement Learning, which progressively evolves task formats and reward functions to match models’ growing capabilities; and (2) Rejected Sampling-based Self-improvement, which maintains the fundamental capabilities of VLMs through selective learning from high-quality examples. Extensive experiments demonstrate that Curr-ReFT achieves state-of-the-art performance across various visual tasks in both in- and out-of-domain settings and benchmarks.

## introduction recent advances in multimodal understanding have led to remarkable vision-language models (vlms), exemplified by openai @xcite @xcite , intervl @xcite , and qwen @xcite series. however, these achievements predominantly rely on massive model scaling (>32b parameters), creating substantial deployment barriers in resourceconstrained environments. this limitation moticurr-rl systematically reformulates input questions into three progressively complex formats. using multimodal math (openr1-8k) as the test case, the pass@k @xcite curves reveal a fundamental trade-off between solution space constraints and reasoning complexity. while supervised fine-tuning (sft) with highquality annotated data @xcite is the dominant training approach for vlms, it poses a critical challenge for smallscale vlms: generalization collapse @xcite @xcite . as evidenced in fig. 1 (a), sft-trained models consistently outperform base models on in-domain data across detection, classification, and multimodal math tasks. however, these gains are accompa-nied by significant performance degradation on out-of-domain (ood) distributions, underscoring the challenge of "ood degradation". this phenomenon aligns with recent theoretical findings @xcite attributing ood degradation to sft's inherent bias toward pattern memorization rather than systematic reasoning @xcite . deepseek r1-zero's @xcite success with group relative policy optimization (grpo) suggests a promising direction for enhancing reasoning through comparative response evaluation. motivated by these advances, we investigate whether rl-based post-training can similarly enhance ood generalization in small-scale vision-language models. comprehensive experiments reveal a consistent pattern: while sft suffers significant ood degradation, rl-based methods maintain robust generalization across diverse visual tasks (fig. 1 (a) ). empirical analysis further demonstrates that rule-based rl reduces sft's susceptibility to visual hallucinations under distribution shifts (more details in sec. 4.2.1). although rl effectively mitigates ood challenges, it encounters 'training bottleneck' in visual reasoning tasks, characterized by minimal policy updates, premature convergence to suboptimal strategies, and repetitive generation of low-quality responses (fig. 1 (b) ). this bottleneck arises from sparse reward @xcite @xcite -tasks with complex solution spaces provide rarely positive feedback, leading to insufficient learning and suboptimal convergence. to address this bottleneck, we propose a structured curriculum reinforcement learning (curr-rl) paradigm that progressively evolves task formats to match models' growing capabilities, inspired by curriculum learning @xcite @xcite . our key insight is that the sparse reward dilemma mainly arises from the vast solution space, hindering the exploration of correct paths, particularly in early-stage training. @xcite . as illustrated in fig. 1 (c ), curr-rl employs a three-phase task reframing and hierarchical reward design, enabling smooth transitions from structured to open-ended formats. this progression begins with binary decisions that reconstruct complex visual reasoning into true/false questions, reducing the solution space dimension for more dense rewards. it then progresses to choice selection formats that introduce partially open elements, while curriculum rl effectively boosts domainspecific visual reasoning, it often compromises general-purpose language capabilities (e.g., commonsense and scientific reasoning), a known tradeoff in rl fine-tuning @xcite . to address this issue, we introduce a rejection sampling-based self-improvement mechanism that preserves general knowledge. built upon the crescent framework @xcite , our method employs an llm-as-judge (e.g., gpt-4o @xcite ) to compare the rl-trained model's responses with reference answers and retain the higher-quality responses. to this end, we propose curriculum reinforcement finetuning (curr-reft). curr-reft comprises two sequential stages: structured curriculum reinforcement learning and rejected samplebased self-improvement. extensive experiments demonstrate curr-reft's state-of-the-art performance on both in-and out-of-domain visual tasks and abundant public benchmarks, with our enhanced small-scale vlms matching the capabilities of much larger counterparts. contributions summary. (1) empirical insight: our experiments reveal that rule-based rl counteracts sft-induced visual hallucinations, significantly improving ood generalization in visual tasks via iterative perception refinement.(2) curr-reft framework: an adaptable two-stage posttraining paradigm that strengthens visual reasoning while preserving fundamental language capabilities; (3) curriculum dataset: a newly constructed 12k-example dataset spanning detection, classification, and multimodal math; (4) empirical results: extensive experiments demonstrate curr-reft's superior performance across multiple benchmarks. ## method in this section, we elaborate curr-reft, comprising two sequential training stages: curriculum reinforcement learning (sec. 3.2), which achieves task progression training through three stages of reward mechanisms, and rejected sample based self-improvement (sec. 3.3), which preserves fundamental capabilities via quality-guided learning. the overall framework is illustrated in fig. 2 . ## conclusion in this paper, we introduce curr-reft, a novel two-stage post-training paradigm that balances domain-specific visual reasoning and general vision-language capabilities. we provide theoretical insights that reinforcement learning improves both reasoning and out-of-domain visual task generalization. we also release a 12k-example curriculum benchmark spanning visual detection, classification, and multimodal math. curr-reft achieves sota on abundant benchmarks, with +5.2% avg. gain in ood visual tasks.