Inefficiencies of Meta Agents for Agent Design (2025)

Abstract: Recent works began to automate the design of agentic systems using meta-agents that propose and iteratively refine new agent architectures. In this paper, we examine three key challenges in a common class of meta-agents. First , we investigate how a meta-agent learns across iterations and find that simply expanding the context with all previous agents, as proposed by previous works, performs worse than ignoring prior designs entirely. We show that the performance improves with an evolutionary approach. Second , although the meta-agent designs multiple agents during training, it typically commits to a single agent at test time. We find that the designed agents have low behavioral diversity, limiting the potential for their complementary use. Third , we assess when automated design is economically viable. We find that only in a few cases—specifically, two datasets—the overall cost of designing and deploying the agents is lower than that of human-designed agents when deployed on over 15,000 examples. In contrast, the performance gains for other datasets do not justify the design cost, regardless of scale.

## introduction agentic systems powered by language models demonstrated remarkable abilities to perform complex tasks and became a transformative force in many domains, including cutting-edge research and development @xcite @xcite , financial services @xcite , and task automation @xcite . until recently, these systems were designed by researchers who built their domain knowledge into their agent architectures. however, a persistent trend in machine learning research, known as the bitter @xcite , suggests that hand-designed solutions are eventually replaced by solutions developed via scalable approaches that leverage search and learning. to this end, recent works have taken the first steps in the direction of automating the design of agentic systems @xcite @xcite @xcite @xcite @xcite . our work focuses on a common class of meta-agents that follow the sample-evaluate-iterate pattern (see figure 1 , algorithm 1) and highlights three challenges. meta learning we begin by examining the assumption that the meta-agent effectively learns from previously discovered agents. our analysis reveals that the meta-agent framework proposed by @xcite does not meaningfully leverage prior designs. in fact, it performs worse than a baseline that ignores prior designs entirely. in contrast, we demonstrate that an evolutionary context curation strategy, where the generation of the next agent is conditioned on the previous best-performing agents (parents), yields improved performance. ## related works our primary reference is adas @xcite , which has introduced meta-agent search with the idea of searching for agents in the code space. mas-gpt @xcite and scoreflow @xcite develop meta-agents by training a model to dynamically generate multi-agent systems for a given query. agentsquare @xcite and archon @xcite explore modular agent architectures and use discrete module recombination to efficiently search design spaces. autoflow @xcite , weak-for-strong @xcite , and adas @xcite use a meta agent that follows the sample-evaluate-iterate paradigm (algorithm 1). other recent meta-agent approaches include multi-agent supernet @xcite , flow @xcite , and aflow @xcite . erol et al. (2025) examined the cost of producing a correct response, which is directly relevant to our economic viability analysis. algorithm 1 meta agent: sample-evaluate-iterate 1: d train # set of training examples 2: f # initial agents library 3: @xmath0, s 0 i ) | f 0 @xmath1: for t in [t ] do 5: â = ϕ(a) # select current context 6: f t ∼ π(• | â) # sample, revise, debug 7: s @xmath2: a.append(f t , s t ) # add to archive 9: # iterate 10: end @xmath3. ( 2024 ), we define an agent as a computer program that takes a question as input and makes language model calls to compute the answer. let f i denote an agent and score s @xmath4, d train ) ∈ r n train be the evaluation vector containing the agent i's evaluation scores for each example in the training dataset d train . the agent f i is represented by code. the archive, a, is a set of discovered agents {f i } and their corresponding evaluations on the training set. we initialize the archive with the agents in the initial agents library, f , and their corresponding evaluations. foot_0 at each iteration, the meta-agent samples a new agent design using a language model, π, conditioned on a curated subset of the current archive, â. the function ϕ implements this curation step. the sampling step is followed by revisions to ensure proper formatting and debugging with execution feedback. finally, the new agent, f t , is added to the archive a. foot_1 algorithm 1 outlines the design procedure. we experiment with three instantiations of context curation (ϕ): cumulative. ϕ c is identity, and the generation of the next agent is conditioned on all the previously discovered architectures, as in @xcite . parallel. ϕ p maps any archive to only the subset that contains 7 agents in the initial library and corresponding evaluation scores. hence, the metaagent ignores the previously designed architectures, effectively parallel sampling the new agents. evolutionary. ϕ e selects a subset of 7 agents with the best evaluation scores (top-k selection strategy) from a to be the parents of the next agent generation. the generation of the next agent is conditioned on this higher-quality subset of the previously discovered architectures at each iteration. tasks and models closely following the prior work @xcite , we evaluate our agentic design setup on 1) mathematical reasoning abilities in a multi-lingual setting, mgsm, @xcite , 2) reading comprehension, drop, @xcite , 3) multi-task problem solving, mmlu, @xcite , and 4) graduate-level science questions, gpqa @xcite . from these datasets, we sample disjoint subsets d train to compute s i , and d test to be used as held-out evaluation. the details of our experimental setup are explained in appendix a.2. all the results we report are averaged across 3 runs. ## conclusion our analysis highlights key trade-offs between (1) final performance and behavioral diversity and (2) performance relative to cost. evolutionary context curation boosts performance but reduces diversity. while meta-agent-driven design can produce costeffective agents in some cases, the performance gains rarely justify the increased design and inference costs, even at scale.