Bridging the Capability Gap: Joint Alignment Tuning for Harmonizing LLM -based Multi-Agent Systems (2025)

Abstract: The advancement of large language models (LLMs) has enabled the construction of multi-agent systems to solve complex tasks by dividing responsibilities among specialized agents, such as a planning agent for subgoal generation and a grounding agent for executing tool-use actions. Most existing methods typically fine-tune these agents independently, leading to capability gaps among them with poor coordination. To address this, we propose MOAT, a Multi-Agent Joint Alignment Tuning framework that improves agents collaboration through iterative alignment. MOAT alternates between two key stages: (1) Planning Agent Alignment, which optimizes the planning agent to generate subgoal sequences that better guide the grounding agent; and (2) Grounding Agent Improving, which fine-tunes the grounding agent using diverse subgoal-action pairs generated by the agent itself to enhance its generalization capablity. Theoretical analysis proves that MOAT ensures a non-decreasing and progressively convergent training process. Experiments across six benchmarks demonstrate that MOAT outperforms state-of-the-art baselines, achieving average improvements of 3.1% on held-in tasks and 4.4% on held-out tasks.

## introduction the rapid advancement of large language models (llms) has significantly transformed the development of intelligent agents capable of reasoning, decision-making, and interacting with complex environments @xcite @xcite . previous work typically involves prompting or fine-tuning a single foundation model on a specific dataset, training the llms how to use external search engines for information retrieval or call web apis for tasks like travel planning @xcite @xcite . recently, to enable llm-based agents to handle more real-world and multi-step tasks, more and more research has increasingly focused on multi-agent systems @xcite @xcite @xcite , which aim to synergize functionality-specialized agents. figure 1 illustrates a commonly-used pipeline, where a multiagent system typically includes a planning agent that decomposes the task into subgoals, followed by a grounding agent that executes these subgoals by invoking appropriate tools, ultimately producing the final solution. despite the progress made by existing multiagent systems, effectively aligning different agents toward holistic performance remains an active research challenge. most existing methods construct specific training data for each agent and train each agent independently. while this decomposition can enhance overall performance, it does not guaran-tee effective collaboration among agents. as illustrated in figure 1 (a), independently trained agents often exhibit varying levels of proficiency, leading to capability mismatches. for example, a planning agent might generate high-level subgoals that are difficult for a weaker grounding agent to understand and execute. conversely, a strong grounding agent might struggle with subgoals generated by a weaker planning agent, leading to errors or inefficient task execution. without explicit mechanisms for adapting to each other's behaviors, these agents struggle to collaborate effectively, resulting in misaligned interactions and coordination failures. in this work, we focus on this plan-ground-execute paradigm since it has been widely adopted in most multi-agent frameworks @xcite @xcite , serving as a general and foundational pipeline. to address the above challenges, we propose moat, a multi-agent joint alignment tuning framework that iteratively alternates between two key stages to achieve alignment in a multi-agent system: (i) planning agent alignment, and (ii) grounding agent improving. unlike prior works that train each agent independently, moat performs multi-agent joint alignment tuning by iteratively and coordinately optimizing the planning and grounding agents. specifically, in the planning agent alignment stage, moat optimizes the planning agent to generate subgoals that better guide the grounding agent in producing correct tool-calling actions. given an input task, we first sample multiple candidate sequences of subgoals from the planning agent. for each subgoal sequence, we then evaluate its effectiveness by measuring the perplexity of the grounding agent in generating correct tool calls conditioned on each sequence. perplexity reflects how well the grounding agent can follow a subgoal, where lower perplexity indicates higher suitability @xcite . using this as a reward signal, we apply the direct preference optimization (dpo) algorithm @xcite to train the planning agent to align with the grounding agent's preferences. in the grounding agent improving stage, we aim to enhance the grounding agent's ability to interpret and act upon subgoals produced by the planning agent. specifically, we reuse the subgoal sequences from the planning agent in the first stage as training data, exposing the grounding agent to realistic settings. for each input task, we use the subgoal-action pairs to train the grounding agent via standard language modeling loss. comparing with relying on ground truth or handcrafted input subgoals, this allows the grounding agent to adapt to the distribution of subgoals produced by the planning agent at practical inference time, thereby improving its robustness and execution accuracy. through theoretical analysis, we demonstrate that the holistic performance of the multi-agent system is improved progressively by alternating the above two stages. we apply moat to several open-source model families (llama, mistral, and qwen) and evaluate it on three types of tasks, i.e., : web, math, and qa, across six benchmarks. the results show that moat consistently outperforms existing baselines, on both in-distribution training sets and out-of-distribution test sets. these validate the effectiveness of our joint alignment framework and demonstrate its strong generalization ability. our main contributions are as follows: (i) we introduce moat, a multi-agent joint alignment tuning framework to jointly optimize interconnected agents, bridging the capability gap between them; (ii) we provide formal analysis proving that the alternating optimization of planning and grounding agents guarantees non-decreasing performance and convergence; and (iii) extensive experiments on both held-in and held-out settings across six benchmarks demonstrate that moat achieves the best performance with 4.4% improvement. agent tuning. agent tuning improves a model's ability to perform downstream tasks by fine-tuning open-source llms using trajectories distilled from stronger models @xcite @xcite . for example, approaches such as agenttuning @xcite , and agentohana @xcite fine-tune smaller models on datasets generated by gpt-series llms. while these improve instruction following and reasoning, single-agent tuning remains limited for complex tasks requiring long-term planning and execution @xcite . to overcome these limitations, frameworks like lumos @xcite and α-umi @xcite propose multi-agent training methods that enable collaboration across functionalityspecialized agents. more recent work like au-toact @xcite further advances this direction by introducing a self-training process where each agent is trained on a dataset generated by itself. however, existing methods often train agents independently, lacking joint optimization to ensure effective coordination. in contrast, our work performs iterative joint tuning to align agents' capabilities for improved cooperation. ## task preliminary a multi-agent system typically consists of three components: (1) a planning agent that breaks down tasks into subgoals, (2) a grounding agent that converts subgoals into executable actions, and (3) an execution module that carries out the actions to get the final answer. given a complex task x, the planning agent, denoted as π p is tasked to decompose it to a sequence of subgoals, formulated as @xmath0. each s i represents a subgoal like "calculate the total number of units in the entire building", contributing to solving the overall task x. next, the grounding agent, denoted as π g , takes the task x, the set of available tools i as well as the decomposed subgoals s as input to generates a sequence of tool calls @xmath1, i, s) = {a i | @xmath2. each a @xmath3, such as "@xmath4. finally, the execution module is responsible for executing the generated tool-call sequence a to accomplish the user task x. ## multi-agent joint alignment tuning the proposed multi-agent joint alignment tuning (moat) framework aims to iteratively align the planning and grounding agents, enhancing the overall performance and coordination of the multiagent system. as illustrated in figure 2 , moat alternates between two stages: (1) planning agent alignment, where the planning agent explores diverse subgoal sequences to guide the grounding agent more effectively, and (2) grounding agent improving, where we reuse the generated sub-goal sequences, improving the grounding agent to better understand them and generate correct actions. through this iterative process, both agents progressively adapt to each other, resulting in more coherent subgoal generation, and more accurate tool calling, towards holistic improvement. ## theoretical analysis in our framework, the planning agent and grounding agent are optimized iteratively. in this section, we provide a theoretical analysis to demonstrate that each optimization step leads to non-decreasing improvements and ultimately ensures the convergence. we start by defining the expected performance of the overall multi-agent system as: here the reward function r(s, a) evaluate the quality of tool-calling action a given sub-goal sequence s. and x indicates the input task. below, we can state the following two lemmas. lemma 5.1. optimizing the planning agent while keeping the grounding agent fixed leads to a nondecreasing expected reward. the planning agent is optimized using dpo, with ppl as the reward signal. the optimization objective can be formalized as: since ppl is negatively correlated with the true reward r(s, a), this is equivalent to maximizing the expected reward: the dpo algorithm guarantees that updates to π p lead to non-decreasing expected rewards when the grounding agent is fixed. thus, we have: t) . this inequality holds because the optimization process aligns the planning agent with sub-goal sequences that facilitate better performance in the grounding agent. lemma 5.2. optimizing the grounding agent while keeping the planning agent fixed leads to a nondecreasing expected reward. the grounding agent is optimized through supervised fine-tuning using pairs (s, a) generated by the planning agent. the corresponding optimization objective is: hence, optimizing the grounding agent improves or maintains the expected reward when the planning agent is fixed. from lemma 5.1 and lemma 5.2, we establish that both optimization steps ensure non-decreasing expected rewards, i.e., e[r] (t+1) ≥ e[r] (t) . additionally, the expected reward e[r] is upperbounded due to the following reasons: (i) the reward function r(s, a) is bounded in practical scenarios; and (ii) the ppl has a lower bound. based on the monotone convergence theorem @xcite , the non-decreasing and upper-bounded nature of {e[r] (t) } ∞ @xmath5. thus, we derive the convergence of overall training process. ## conclusion in this work, we present moat, a novel joint alignment tuning framework designed to harmonize the collaboration between planning and grounding agents in llm-based multi-agent systems. by iteratively optimizing the planning agent to generate subgoals that align with the grounding agent's capabilities and enhancing the grounding agent's adaptability to diverse subgoal sequences, moat effectively bridges the capability gap caused by independent training. both theoretical analysis and extensive experimental results demonstrate the superiority of moat. we suggest that future work can explore the integration of visual models to expand the capability boundary of agents, as well as integrate more specialized agents, such as tool retrieval and reflection modules, to expand the versatility and efficiency of the system.