Abstract
Automated agent architecture search demonstrates potential for improving embodied agent performance while revealing challenges related to optimization signals, local optima, and credit assignment in simulation-based training.
Embodied agents are typically built as hand-designed compositions of perception, memory, planning, and action modules. This modularity exposes a large architectural design space, but current systems still rely on researcher intuition to choose where information is stored, how observations are processed, and how model calls are connected. Agent Architecture Search (AAS) automates such design for text-domain agents, but has not been systematically evaluated on perceptual embodied agents through simulator rollouts. We study this transfer. We introduce AgentCanvas, a typed-graph runtime that hosts embodied executors as editable node-and-wire programs with simulator-aware execution and episode-level logs, and KDLoop, a coding-agent search procedure that cycles through proposal, critique, experiment, and distillation, with triggered reflection after stalls. We evaluate three AAS variants across four embodied executors spanning vision-language navigation, embodied question answering, and language-conditioned manipulation. The resulting 3x4 matrix shows that architecture-level search can produce deployable and directional success-rate gains on embodied tasks, while one apparent high-scoring candidate is rejected as leak-bearing. At the same time, the experiments expose constraints that are muted in text-domain AAS: optimization signals can be masked by rollout noise, search can become trapped in local edit basins, and episode-level credit assignment only partially emerges even when detailed logs are available. These results characterize both the promise and the current limits of automated architecture search for embodied agents.
Community
Can embodied agent design be searched instead of hand-built?
Unlike a policy trained end-to-end, an embodied agent is
assembled: perception, memory, planning, and action wired
together by hand — module by module, for one benchmark at a time.
That hand-wiring is an explicit design space. So we ask: can a
search explore it instead of a researcher? We take the first step.
Two pieces:
- AgentCanvas — a typed node-and-wire runtime that hosts a
published embodied agent (MapGPT, SmartWay, ExploreEQA,
VoxPoser) as an editable graph that runs and logs itself inside
a stateful simulator. - A coding agent carries the search: it reads the codebase,
proposes a graph edit, implements it, runs the rollout, reads the
result — loop after loop. Our variant KDLoop cycles Think →
Critic → Experiment → Distill → Reflect; we also port ADAS and
AFlow into the same harness for a fair comparison.
Across a 3×4 optimizer×executor matrix — spanning VLN, EQA, and
VLA (12 cells) — search finds rerun-confirmed, three-pass gains:
- MapGPT · VLN — 46.9 → 54.5% SR (+7.6)
- ExploreEQA · EQA — 43.0 → 47.7% (+4.7)
- SmartWay · VLN — 29.7 → 33.7% (+4.0)
- VoxPoser · VLA — 9.0 → 12.9% (+3.9)
Functional behavior changes — not the cosmetic workflow
reshuffling recent text-domain critiques warn about.
And then the embodied setting bites back — which is half the
paper:
- One apparent +9.0 SmartWay "win" was a leak 🕵️ — the search
had wired the evaluator's ground-truth signal straight into the
planner. Only the variant that reads the episode logs, not just
the score, caught it. - Three limits text-domain AAS mutes: rollout noise masks good
edits, search stalls in local edit basins, and
episode-level credit assignment only partially emerges — even
with full logs.
Not a leaderboard paper — the first systematic study of moving
Agent Architecture Search onto real, perceptual embodied agents
validated through simulator rollouts, plus an honest map of what
that search must overcome. AgentCanvas ships as a standalone
visual platform, not just a paper artifact.
💻 Code + docs (Apache-2.0):
https://github.com/jianzhou0420/AgentCanvas
📄 Paper project page:
https://jianzhou0420.github.io/src/works/AgentCanvas/paper.html
🌐 AgentCanvas project page:
https://jianzhou0420.github.io/src/works/AgentCanvas/index.html
Get this paper in your agent:
hf papers read 2606.30111 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper