LSRL : Process-Supervised GRPO on Latent Recurrent States Improves Mathematical Reasoning (2025)

Abstract: Latent-recurrent language models solve tasks by iteratively refining hidden states rather than emitting chain-of-thought tokens, yet the opacity of those hidden trajectories hinders credit assignment and limits mathematical reasoning accuracy. We propose Latent-State Supervised Reinforcement Learning (LSRL), a process-supervised variant of Guided Reward Policy Optimization (GRPO) that delivers dense rewards at every latent step. We decode each recurrent depth of a 3.5-billion-parameter Huginn model and score the partial solutions with a GPT-4.1-nano grader aligned to final-answer correctness. Using LoRA adapters, we update the policy on a single NVIDIA L40S GPU with only 500 GSM-8K training problems. Relative to the depth-8 supervised Huginn baseline, LSRL improves absolute accuracy by +4.27 points on GSM-8K and +2.06 points on MathQA. These results demonstrate that rewarding latent steps provides an efficient route to stronger mathematical reasoning in latent-recurrent language models.

## introduction latent-recurrent language models (lr-lms) refine internal states through recurrent processing rather than generating explicit chain-of-thought tokens. the recent huginn model @xcite demonstrates that increasing the recurrent depth from @xmath0, effectively trading parameters for compute. however, the same 3.5-billion-parameter model achieves only 13.5% on the gsm-8k math benchmark at @xmath1, far below its logical performance, and the deeper @xmath2.9%. why does increased depth yield such limited gains? we argue that the bottleneck is sparse credit assignment: huginn's supervised finetuning pipelines apply a single reward to the final answer, ignoring the quality of intermediate latent states. by contrast, process supervision, which rewards every step of a token-level derivation, has recently improved mathematical reasoning in chain-of-thought models @xcite . to date, however, process rewards have never been applied to latent states, as decoding and grading all r hidden snapshots appears prohibitively expensive. our approach. to address this limitation, we introduce latent-state supervised reinforcement learning (lsrl), a critic-free grpo variant that attaches dense rewards to every latent depth of huginn. a lightweight gpt-4.1-nano grader scores each partial derivation, and lora adapters @xcite update the policy efficiently. contributions. the remainder of this paper is organized as follows. section 2 reviews latent-recurrent lms, math-oriented rl, and process supervision. section 3 details the lsrl framework. sections 4-5 present experimental setup, results, and analysis. section 6 discusses limitations and future work. ## conclusion and future work we introduced latent-state supervised reinforcement learning (lsrl), a process-supervised variant of grpo that delivers dense rewards at each latent iteration of recurrent language models. our approach addresses a key limitation of latentrecurrent models: while they can achieve impressive reasoning capabilities with fewer parameters, their opaque hidden trajectories hinder effective credit assignment, especially for mathematical reasoning tasks. our work makes several contributions to latent reasoning in language models. first, we developed a novel framework for process-supervised rl that operates on latent states rather than explicit tokens, creating the first process-supervised approach for latent recurrence. second, we introduced technical innovations that make this approach practical: a one-pass hidden-state cache that fully decodes every latent depth in a single forward/backward pass, and lora adapters for efficient fine-tuning of latent-recurrent rl. finally, using only 500 gsm-8k training problems and a single gpu, we demonstrated substantial improvements of +4.27 points on gsm-8k and +2.06 points on mathqa, approaching the performance of models with 4× the compute requirements. our results demonstrate that latent-recurrent architectures offer a promising alternative path to scaling reasoning capabilities in language models. while most approaches focus on either increasing parameter count or extending inference through chain-of-thought tokens, lsrl enables models to scale through test-time computation in the latent space. this provides several advantages: (1) reduced memory requirements during training and inference, and (2) no need for specialized training data containing intermediate reasoning steps. future directions. there are several promising directions for extending this work: we believe that lsrl represents an important step toward more efficient mathematical reasoning in language models. by aligning the optimization process with the recursive structure of latent-recurrent models, we achieve performance that would typically require significantly more parameters or deeper recurrence depth. this suggests that process-level supervision in the latent space is a promising direction for developing more capable yet efficient reasoning systems.