Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning (2025)

Abstract: Instruction-fine-tuned large language models (LLMs) under 14B parameters continue to underperform on natural language understanding (NLU) tasks, often trailing smaller models like BERT-base on benchmarks such as GLUE and SuperGLUE. Motivated by the success of reinforcement learning in reasoning tasks (e.g., DeepSeek), we explore Proximal Policy Optimization (PPO) as a framework to improve the NLU capabilities of LLMs. We frame NLU as a reinforcement learning environment, treating token generation as a sequence of actions and optimizing for reward signals based on alignment with ground-truth labels. PPO consistently outperforms supervised fine-tuning, yielding an average improvement of 6.3 points on GLUE, and surpasses zero-shot and few-shot prompting by 38.7 and 26.1 points, respectively. Notably, PPO-tuned models outperform GPT-4o by over 4% on average across sentiment and natural language inference tasks, including gains of 7.3% on the Mental Health dataset and 10.9% on SIGA-nli. This work highlights a promising direction for adapting LLMs to new tasks by reframing them as reinforcement learning problems, enabling learning through simple end-task rewards rather than extensive data curation. Our code is available at https://github.com/coder-qicao/RL4GLUE.

## introduction large language models (llms) @xcite @xcite have revolutionized natural language processing (nlp) with their powerful text generation capabilities @xcite . pretrained on large-scale unlabeled corpora, llms can generate coherent and contextually relevant content. using promptbased strategies like zero-shot and few-shot prompting (brown, 2020), these models can address a * equal contribution wide range of downstream tasks without taskspecific fine-tuning. however, when applied to instruction-fine-tuned llms under 14b parameters foot_0 -such as llama2-7b-chat-hf-these methods often underperform on natural language understanding (nlu) tasks compared to encoder-only models like bert @xcite , which consistently excel on benchmarks such as glue @xcite and superglue @xcite . for instance, our evaluation of llama2-7bchat-hf shows that zero-shot prompting with taskspecific prompts yields an average performance of 46.1 across all glue datasets, while few-shot prompting improves performance to 58.7-both significantly trailing bert-base's 79.6, as shown in to enhance nlu capabilities of llms, we investigate reinforcement learning (rl)-based finetuning approaches. motivated by recent work such as deepseek @xcite , which demonstrates the utility of reward-driven optimization for improving reasoning abilities, we explore the use of proximal policy optimization (ppo) @xcite to align model outputs with taskspecific objectives. while standard fine-tuning (sft) is commonly used to adapt llms to downstream tasks, we find it insufficient for nlu-often underperforming even smaller encoder-only models like bert-base. in contrast, we use ppo to enhance llm performance by framing nlu as a reinforcement learning problem, as shown in fig 1 . the sequence of input tokens up to timestep t-1 represents the state s t , and the token generated at timestep t is treated as the action a t . after generating the full response, a heuristic extracts the predicted answer, which is compared to the ground-truth label to assign a scalar reward r. ppo then updates the model to maximize to address this performance gap, we recast the task of nlu adaptation as an rl problem, fine-tuning the model using ppo. the input consists of a prompt and a query. in the standard setting (top path), the base model, with zero-shot/few-shot prompting, struggles to generate correct answers. our approach selectively updates lightweight lora layers, using ppo to optimize a reward signal based on task correctness. this optimization encourages performance gains while constraining deviation from the base policy. we find that ppo fine-tuning substantially outperforms standard sft. pre-trained llms possess broad linguistic knowledge, spanning syntactic and semantic structures, acquired from large-scale text corpora. we show that reinforcement learning, specifically ppo, can refine this general understanding to better align with task-specific nlu objectives. similar patterns are observed in deepseek, where chain-of-thought pre-training enhances reasoning capabilities, and subsequent rl-based fine-tuning further improves performance. building on this insight, our findings suggest a promising direction: adapting llms to new tasks by formulating them as reinforcement learning problems. when models are sufficiently pre-trained, task alignment may be achieved without additional labeled data-requiring only a well-defined reward function over the outputs. ppo can then optimize the model toward high-reward behaviors. this approach offers a scalable, label-efficient alternative to conventional supervised fine-tuning through reward-driven adaptation. ## method to enhance the performance of llms on nlu tasks, we adopt two distinct fine-tuning methods. the first approach involves supervised fine-tuning, where the input consists of a concatenation of the task-specific prompt, query and the ground truth answer, with the model optimized using the nexttoken prediction objective. the second approach utilizes ppo, framing response generation as a reinforcement learning problem. in this setup, the sequence of input tokens until timestep t -1 represents the state s t , and each token generated at timestep t is treated as an action a t . after generating the entire sequence, a heuristic-based process extracts the final answer from this generated sequence, and is compared to the ground truth. ppo is then employed to optimize the model by maximizing the cumulative reward derived from this comparison. to reduce computational complexity, we fine-tune lora layers instead of the full model. we refer the readers to appendix a and b for preliminaries on ppo. ## conclusion prompting-based approaches, including zero-shot and few-shot prompting, are commonly used to adapt llms to downstream tasks. however, our experiments show that when applied to llama2-7b-chat-hf, these methods underperform on nlu benchmarks such as glue and superglue (