Beyond Numeric Rewards: In-Context Dueling Bandits with LLM Agents (2025)

Abstract: In-Context Reinforcement Learning (ICRL) is a frontier paradigm to solve Reinforcement Learning (RL) problems in the foundation-model era. While ICRL capabilities have been demonstrated in transformers through task-specific training, the potential of large language models (LLMs) out of the box remains largely unexplored. This paper investigates whether LLMs can generalize cross-domain to perform ICRL on the Dueling Bandits (DB) problem, a stateless preference-based RL setting. We find that top-performing LLMs exhibit a notable zero-shot capacity for relative decision-making, which translates to low short-term weak regret across all DB environments by quickly including the best arm in duels. However, an optimality gap still exists between LLMs and classic DB algorithms in terms of strong regret. LLMs struggle to converge and consistently exploit even when explicitly prompted to do so, and they are sensitive to prompt variations. To bridge this gap, we propose an agentic-flow framework—LLM with Enhanced Algorithmic Dueling (LEAD)—which integrates off-the-shelf DB algorithm support with LLM agents through fine-grained adaptive interplay. We show that LEAD inherits theoretical guarantees from classic DB algorithms on both weak and strong regret. We validate its efficacy and robustness even with noisy and adversarial prompts. The design of such an agentic framework sheds light on how to enhance the trustworthiness of general-purpose LLMs generalized to in-context decision-making tasks.

## introduction transformers pretrained with interactive datasets have led to the emergence of in-context reinforcement learning (icrl) @xcite , where models can infer reinforcement learning (rl) tasks from interaction his-( , ) the reward is 𝗋(t) tories as context and make effective decisions without parameter updates. through trial and error, these models can self-improve their policies purely in-context. recent investigations into llms' icrl capabilities in environments with numeric rewards have reported notable failure cases, e.g., llm agents being vulnerable to adversarial loss functions and suffering from high regret compared to classic algorithms such as follow-the-regularized-leader (ftrl) @xcite , and exhibiting failures in exploration within multi-armed bandit (mab) problems (krishnamurthy et al.). even with inference-time algorithm guidance, an optimality gap persists between llms and classic mab algorithms @xcite . these results highlight the need for further research in non-trivial algorithmic interventions to elicit desirable icrl behavior in llm agents. the failure cases encountered by llms may be attributed to intrinsic difficulties in processing numeric rewards, especially in tasks where patterns are difficult to express in natural language. recent findings have pointed out that llms often struggle with simple numerical comparisons (e.g., incorrectly judging 13.11 to be larger than 13.8), and there has been a notable lack of emphasis on evaluating the relative comparisons among the decisions they generate. figure 1 shows a toy example illustrating the in-context interaction between an llm agent and different environment settings. to disentangle the complexities introduced by numerical rewards, this paper focuses on the problem of dueling bandits (db) @xcite , a stateless preference-based reinforcement learning setting @xcite that extends the classic mab model by querying for preference feedback between selected pairs of arms to identify the best one. in db, the agent learns through binary outcome (win or lose) of a noisy comparison between the two selected arms. this setup is particularly useful when eliciting explicit feedback is challenging or when the feedback is inherently comparative, such as the taste of food and product attractiveness @xcite . db has attracted significant attention due to its applicability in information retrieval @xcite , recommendation systems @xcite , and online ranker evaluation @xcite . we frame our investigation with the following question: are llms effective in-context agents for solving the problem of dueling bandits? the db problem poses distinctive challenges as a relative decision-making instance, particularly due to the sparse nature of the relative rewards. this sparsity complicates the in-context decisionmaking process, as it restricts the feedback obtained from interactions, introducing a level of difficulty not typically seen in conventional bandit problems. even though reduction from db to standard mab exists @xcite , it remains unclear how llms would perform in db with preference feedback rather than numeric rewards. there are conceptual differences between them, similar to those between reinforcement learning from human feedback (rlhf) @xcite and standard rl, where impossibility results can be found in @xcite . while task-specific training of large sequence models can yield promising icrl results, it is often impractical due to the substantial computational resources required. similar to the settings in @xcite @xcite , we evaluate the emergent zero-shot abilities @xcite of icrl in general-purpose llms under the dueling bandit problem, without re-training or fine-tuning. we summarize our main results below. evaluation of llms' emergent zero-shot abilities of in-context db. we go beyond numeric rewards to evaluate the performance of llm agents in terms of both strong and weak regret for making decisions in db by comparing against various baseline db algorithms via a case study. we found that our top-performing llm has the zero-shot relative decision-making ability sufficient to achieve low weak regret in db, which significantly differs from that in classic mab settings @xcite . notably, gpt-4 turbo can serve as an effective decision-maker for dueling bandits in terms of weak regret, quickly selecting the best arm in duels with low variance across a range of instances. however, consistent with @xcite , we found that an optimality gap exists between llms and classic db algorithms in terms of strong regret. llms' performance is hindered by over-estimation bias in the exploration stage and lack of convergence criterion in the exploitation stage. this highlights the need for non-trivial algorithmic interventions to achieve cross-domain generalization. effective and robust agentic flow framework for in-context db. to address the identified optimality gap and enhance the trustworthiness of in-context llm agents in db tasks, in section 4.1, we propose an agentic flow framework, llm with enhanced algorithmic dueling (lead) that integrates off-the-shelf explore-then-exploit db algorithms with llm agents. this framework enables the fine-grained adaptive interplay between db algorithms and in-context llm agents, pushing forward from the naive algorithm-guided support used in @xcite . as an illustrative example, we demonstrate how the interleaved filter2 (if2) algorithm can be incorporated with llm agents in this framework. we show that lead has theoretical guarantees, with experiments demonstrating its efficacy and robustness across various prompting scenarios. ## preliminaries in this section, we briefly introduce the problem of dueling bandits (db) and establish the necessary notation for this paper. additional useful definitions can be found in appendix b.3.1. ## llms as standalone in-context decision-makers to evaluate the llms' efficacy for solving db problems in-context, in this section, we use llms as standalone decision-making agents and compare them with classic db algorithms. our evaluation is two-fold: first, in figures 2 and 9 , we compare the performance of llms and classic algorithms in terms of the strong and weak regret (see eq.( 3 ) and eq.( 4 ), with standard deviation). second, we delve into the experimental results and analyze the success and failure modes of llm agents. ## conclusion this paper evaluates llms as in-context decisionmakers for context-free dueling bandits (db) with a condorcet winner, offering the first systematic insights into their strengths and limitations. our findings reveal that llms' decision-making in db, driven by linguistic priors, achieves exceptional weak regret performance across both transitive and intransitive environment instances in the short-term. however, llms lack the necessary criteria for convergence and long-term generalization to hard scenarios, leading to an optimality gap between llms and classic db algorithms in terms of strong regret. to bridge this gap, we propose lead, an agentic flow framework that integrates off-the-shelf db algorithms with llm agents through fine-grained adaptive interplay. this framework provides theoretical guarantees and ensures robust performance even under noisy and adversarial prompts. moving beyond the limitations of traditional numeric rewards, it sheds light on how language-based reasoning can be robustly generalized from words to actions. 6 limitations this work primarily focuses on the fundamental context-free dueling bandit (db) setting, exploring the performance and generalization of large language models (llms) within this relative decision-making framework. the scope of this work is limited to this specific setting, and does not address the complexities introduced by other types of db environments. the following limitations are noteworthy: (i) the exploration of llms in conjunction with other ongoing regret-minimization algorithms is not considered here; (ii) the performance of llms under alternative winner definitions, such as borda and neumann winners is not explored; (iii) the paper does not examine llms' behavior in other db settings, such as contextual dueling bandits, multi-dueling bandits, and adversarial dueling bandits. expanding the scope to solve these limitations is an important direction for future work.