Papers
arxiv:2606.17053

Context-Aware RL for Agentic and Multimodal LLMs

Published on Jun 15
Β· Submitted by
py xu
on Jun 19
Authors:
,
,
,
,
,

Abstract

ContextRL enhances long-horizon reasoning and multimodal performance through reinforcement learning that rewards context selection for supporting query-answer pairs, achieving improvements over standard methods on diverse benchmarks.

Large language models (LLMs) often fail when answering requires identifying a small but decisive piece of evidence within a long or complex context, such as a single line in a tool trace or a subtle detail in an image. We propose ContextRL, a context-aware reinforcement learning (RL) method that improves long-horizon reasoning and multimodal performance through an indirect auxiliary objective. Instead of supervising only the final answer, ContextRL presents the model with a query, an answer, and two highly similar contexts, and rewards it for selecting the context that supports the query--answer pair, thereby encouraging fine-grained grounding. We construct contrastive context data in two domains: for coding agents, trajectories serve as contexts, yielding 1k pairs built via condition filtering; for multimodal reasoning, images serve as contexts, yielding 7K pairs built via generative editing and similarity search. ContextRL achieves average gains of +2.2% over standard GRPO on 5 long-horizon benchmarks, and +1.8% across 12 diverse visual question answering benchmarks. To disentangle the effect of the proposed objective from that of additional data, we compare against data-augmentation baselines that repurpose the same contrastive contexts as standard query--context--answer examples. These baselines provide little to no improvement, showing that the gains arise from the proposed context-selection objective rather than from the contrastive data alone.

Community

Paper author Paper submitter

Context-Aware RL for Agentic and Multimodal LLMs

πŸ‘‰ LLMs often fail not because the answer is impossible, but because they miss the one decisive clue hidden in a long trace or image.

πŸ”₯ We introduce ContextRL: RL that teaches models to identify which context actually supports an answer.

βœ… +2.2% on 5 agentic benchmarks
βœ… +1.8% across 12 VQA benchmarks
βœ… Works for coding agents & multimodal reasoning
βœ… Same contrastive data, but better objective β€” not data augmentation

🧠 The key idea: don’t only reward the final answer. Reward the model for grounding it in the right evidence.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2606.17053
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2606.17053 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2606.17053 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2606.17053 in a Space README.md to link it from this page.

Collections including this paper 1