Toward Optimal LLM Alignments Using Two-Player Games (2025)

Abstract: Alignment of large language models (LLM) is a process that ensures the model’s responses to user prompts align with human intentions and social values. This optimization typically relies on pre-collected prompts. The collection of these prompts often either requires careful human interventions or proves to be difficult to have a good coverage over all scenarios an LLM can improve over . To address this issue, we propose an alignment method based on a two-agent game, consisting of an adversarial agent and a defensive agent. The adversarial agent’s task is to generate prompts that expose the deficiencies of the defensive agent. At the same time, the defensive agent improves its performance on the prompts generated by the adversary based on feedback from the reward model. This iterative process is repeated to enhance the model’s performance. We theoretically demonstrate that, under mild assumptions, this iterative alignment process converges to a Nash equilibrium by both agents. Learning in this competitive environment results in policies with better generalization capabilities. We demonstrate the advantage of our framework using extensive experiments.

## introduction large language models (llms), such as chatgpt @xcite , claude (anthropic, 2024), and others, have achieved great success due to their remarkable generalization and versatility. one crucial component of llm development is alignment @xcite @xcite , which ensures llms can follow instructions, understand human intention, and align with social values. performing the alignment of llms requires the preparation of a set of prompts. the traditional alignment method optimizes the model's response on precollected prompts, which are mostly contributed by human labelers and could fail to cover all task types. later, several methods have been proposed to expand the scope of prompts used, including based on difficulty @xcite , paraphrase @xcite , and selfinstruct @xcite . nonetheless, these methods are often rule-based and do not customize or adapt their design to the capabilities of aligning llms, i.e., identify prompts that the aligning llm struggles at responding to. furthermore, using a static prompt dataset may lead to saturation of llm performance due to the loss of discernment by the reward model. therefore, a more dynamic and adaptive approach is necessary for llm alignment to improve its generalization. to address these limitations, we introduce a novel framework inspired by the tutor-student model of human learning, conceptualizing the alignment process as a two-player game. in this framework, an adversarial agent (adversarial llm) and a defensive agent (defensive llm) engage in iterative interactions to enhance both their performances. specifically, the adversarial llm, acting as the tutor, learns to automatically generate prompts that challenge and reveal the weaknesses of the defensive llm. meanwhile, the defensive llm, functioning as the student, is tasked with adapting and improving its responses to these adversarially generated prompts. our framework is grounded in research on learning in competitive multi-agent environments @xcite . this approach fosters a natural curriculum of increasing complexity, allowing both agents to develop progressive behaviors that surpass the inherent complexity of their training environment. figure 1 illustrates our proposed framework using two players. in pursuit of a more robust and comprehensive approach to building the adversarial agent, we also introduce a novel mechanism to incorporate diversity constraints based on bleu scores @xcite and sentence embeddings @xcite . by integrating these diversity constraints, we successfully prevented the adversarial agent from converging prematurely to a narrow set of effective prompts, thereby expanding the coverage of potential vulnerabilities within the llm. theoretically, we demonstrate that this iterative adversarial alignment process converges to a nash equilibrium between the adversarial and defensive agents. this equilibrium signifies a state where neither agent can unilaterally improve their strategy, implying a more comprehensive training process that leads to better coverage of prompts for alignment. our experiments, conducted in scenarios involving harmful inputs and jailbreak settings, validate the effectiveness of the proposed method. the results show that our approach not only enhances the generalization capabilities of the agents but also ensures that both parties in the interaction are thoroughly trained. as a byproduct, in addition to creating a generalizable and well-aligned defensive llm, our adversarial agent also serves as an adaptive red teaming partner, continuously generating challenging prompts to enhance the alignment of the defensive llm. ## preliminary in this section, we briefly recap the basics of llm and the standard rlhf workflow to establish the necessary notations and conceptual framework for our contributions. consider @xmath0, x (2) , . . . , x (m ) ) ∈ x as the given prompt, where x (k) represents the k-th token in the prompt. the goal of the large language model is to generate a response @xmath1, y (2) , . . . , y (n ) ) ∈ y in an auto-regressive manner, governed by the following conditional probability distribution: here, x and y represent the sets of all possible prompts and responses, respectively. the reinforcement learning from human feedback (rlhf) is a widely adopted framework to align an llm behavior to comply better with human preferences. this process involves three main steps: 1) supervised fine-tuning, 2) reward modeling, and 3) rl-based policy optimization. supervised fine tuning. rlhf typically begins with supervised fine tuning (sft), which fine-tunes a pre-trained llm through supervised learning on high-quality samples from downstream tasks. the resulting model is denoted as π sft . reward modelling. the second phase of rlhf involves developing a reward model r(•, •) that reflects human preferences, utilizing annotated data d @xmath2, y c , y r )}, where y c and y r represent the chosen and rejected responses to the prompt x. for instance, in response to a malicious prompt seeking illegal information, the preferred reaction would be to refuse to answer rather than to comply. one widely-adopted objective is to minimize the negative log-likelihood of the bradley-terry (bt) model @xcite , so as to assign higher rewards to the chosen response y c over the rejected response y r : ## game-theoretical preference optimization (gpo) inspired by the tutor-student model of human learning, we aim to create a dynamic learning environment for llms, featuring iterative interactions between an adversarial and a defensive agent. the adversarial agent, serving as the tutor, evaluates past errors and current performance of the defensive agent to dynamically identify and exploit potential weaknesses. in turn, the defensive agent, functioning as the student, continuously adapts and strengthens these identified vulnerabilities. this iterative cycle is repeated to consistently improve performance. ## experiments in this section, we aim to evaluate gpo in safety scenarios, focusing on both general conversation and jailbreak contexts. our objective is to assess whether alignment through two-player games can result in: (1) a more capable adversarial agent that produces diverse and effective attack prompts; and baselines. for evaluation of both the safety of the defensive agent and the attack capabilities of the adversarial agent, we compare the following methods: sft: an adversarial or defensive agent that has only undergone supervised fine-tuning. paraphrase: paraphrasing adversarial prompts experimental setup. for all methods, we utilize the prompts from the anthropic's red teaming @xcite for training, and conduct evaluations as follows. ## related work we primarily focus on discussing the most relevant lines of work, self-play in rlhf, in this section. a detailed discussion of other related works on llm alignment and safety alignment can be found in appendix c. ## conclusion in this work, we introduced a novel framework for aligning llms by conceptualizing the process as a two-player game between an adversarial agent and a defensive agent. through iterative interactions, the adversarial agent learns to generate diverse and challenging prompts to uncover the weaknesses of the defensive llm, while the defensive llm adapts and improves its responses. by incorporating diversity constraints and demonstrating convergence to a nash equilibrium, our approach enhances the generalization capabilities of both agents and ensures thorough training. our experiments validate the effectiveness of the proposed method in scenarios involving harmful inputs and jailbreak settings. our solution does require training two separate llm agents, and this work primarily focused on prototyping our idea using safety-related tasks. in the future, we aim to extend the scope of our alignment framework to address the challenges that arise in other domains. specifically, we hope to investigate the application of our approach in helpfulness and reasoning.