Agent Lumos: Unified and Modular Training for Open-Source Language Agents (2024)

Abstract: Closed-source agents suffer from several issues such as a lack of affordability, transparency, and reproducibility, particularly on complex interactive tasks. This motivates the development of open-source alternatives. We introduce Lumos, one of the first frameworks for training open-source LLM-based agents. Lumos features a learnable, unified and modular architecture with a planning module that learns high-level subgoal generation, and a grounding module trained to translate these into the actions using various tools in the execution module. The design allows for modular upgrades and wider applicability to diverse interactive tasks. To foster generalizable agent learning, we collect large-scale, unified, and high-quality training annotations derived from diverse ground-truth reasoning rationales across various complex interactive tasks. On 9 datasets, Lumos exhibits several key advantages: (1) Lumos excels multiple larger open-source agents on the held-out datasets (unused for training) for each task type. Lumos even surpasses GPT agents on QA and web tasks; (2) Lumos outperforms open-source agents produced by chain-of-thoughts and unmodularized integrated training; and (3) Lumos effectively generalizes to unseen tasks, outperforming 33B-scale agents and domain-specific agents. Code and data will be released.

## introduction language agents execute actions and interact with external tools or environments, in service of a goal. they have evolved into crucial elements of ai systems targeted at solving complex interactive tasks. these tasks often require agents to perform longhorizon planning and interactive reasoning, and can range from qa @xcite , to web tasks @xcite , 2023), math @xcite , and multimodal reasoning @xcite . prior agent frameworks @xcite @xcite @xcite have primarily relied on closedsource large language model (llm) apis such as @xcite . though powerful, they can be prohibitively expensive, particularly for tasks with long contexts such as web tasks (which include encoding long html code). furthermore, the lack of transparency in closed-source llms hinders scientific understanding of their architectures and effectiveness, and provides limited reproducibility, and controllability over their behavior. we argue that over reliance on closed-source llm-based agents is not conducive to the growth of research on language agents. in this paper, we propose lumos, a generalizable language agent framework via unified, modular, and open source training. lumos employs a unified and modular architecture broadly applicable to complex interactive tasks: a planning module , a grounding module , and an execution module . the planning module learns to decompose diverse complex tasks into a sequence of high-level subgoals. the grounding module is trained to communicate with the planning module and translate its generated subgoals into the actions that can be executed through a collection of tools in the execution module. lumos design allows for easy module upgrades to enhance new task planning, novel action grounding and supplementing new tools, without impacting the others. to tackle the tasks through the agent modules, we propose two interaction formulations for implementing the language agents, lumos-onepass (lumos-o) and lumos-iterative (lumos-i). outlined in fig. 2 , lumos-o is an efficient formulation that generates all the subgoals and actions through a single inference call, accomplishing the task in a one-pass manner. lumos-i is an iterative formulation that generates one subgoal at a time based on its previous execution results and environment updates, thereby enabling an adaptive agent. in addition, lumos utilizes a unified data format that encompasses multiple task types, thereby enabling the proposed agent framework to conveniently support a range of interactive tasks. these include, but are not limited to: question answering, mathematics, coding, web browsing, multimodal reasoning, and text games. to obtain high-quality annotations for training lumos, we leverage the ground-truth rationales in existing benchmarks across various task types, and convert them into a unified format ( §3). this conversion is achieved with the aid of strong llms, ensuring that the converted annotations adhere to a universally applicable format consistent with our modular design. our proposed annotation conversion method results in around 56k multi-task multi-domain agent training annotations, one of the largest open-source resources for agent fine-tuning. the training annotations could serve as a resource for universally enhancing any open-source llms with agent capa-bilities. our evaluation demonstrates that lumos provides improved or comparable performance with gpt-based or larger open-source agents across various complex interactive tasks that are commonly used for agent evaluation, including qa, web, math, and multimodal tasks. we summarize our contributions and results as follows: general agent framework with high-quality data. we introduce an open-source agent learning framework that trains llms with unified data, aimed at unifying complex interactive tasks and enhancing generalization on unseen tasks with new environments and actions. we hope our framework and annotations can facilitate future research in developing open-source language agents. competitive performance across tasks and agent training formulations. lumos outperforms a great number of open-source agents on the lumos held-out datasets unused in lumos training data across the four training task types. lumos even surpasses gpt-based agents in web and qa tasks. specifically, lumos shows a 5.0% enhancement over gpt-4 on mind2web, and 4.1% and 3.5% llm accuracy foot_0 improvement on hotpotqa over the react and rewoo agents fully based on gpt-3.5-turbo, respectively. furthermore, we observe that lumos training formulations outperform other potential agent training methods, such as chain-of-thoughts and integrated training, which instruct a single module to both plan and ground. cross-task generalization. we evaluate lu-mos on two unseen tasks, webshop @xcite , a text game for online shopping, and intercode sql @xcite , an interactive code generation task. lumos even surpasses 30bscale agents, especially by nearly 20 reward points on webshop. lumos also delivers a consistent reward improvement over domain-specific agents. this suggests that lumos can generalize across tasks, hinting at potential benefits for a wide spectrum of language agent applications. ## lumos: a modular open-source llm-based agent framework we introduce the overall design and two formulations for developing agents within this framework. ## learning to plan & ground with open-source llms to guide planning and grounding modules to generate subgoals and valid low-level actions under our specified action interfaces, we fine-tune the two modules to produce the expected outputs. training the modules requires the supervisions consisting of high-quality tasks, subgoals, and low-level actions. to equip smaller llms with instruction-following ability, prior works leverage methods such as self-instruct @xcite to synthesize training tasks and inputs, and directly generate ground-truth task outputs based on its created tasks. however, these methods are not suitable for generating high-quality annotations for training agents. for example, given a web browsing task in mind2web, gpt-4 only achieves around 20% step success rate @xcite when completing the task. relying on such methods to generate complex interactive task annotations may degrade the annotation quality. instead of creating annotations with llms from scratch, we exploit llms as a "style transfer" tool to convert ground-truth reasoning steps in existing benchmarks into the expected format in lumos formulations. there are a considerable number of the datasets annotated with either human-written solutions or structured action sequences foot_1 . for example, prm800k @xcite is a math dataset containing the natural language solutions interleaved with formulas; strategyqa @xcite are a qa dataset with decomposed questions, supporting facts, and relevant wikipedia paragraph indices; mind2web includes ground-truth action sequences, etc. they provide llms with fundamental information that sufficiently contributes to the annotation conversion. next, we introduce 1) how we prompt llms to obtain the subgoal and action supervisions for training modules; 2) how to organize the subgoals and actions into the conversational forms aligning with lumos formulations; 3) how we train the modules with the final annotations. ## experiments we begin with the details of our experimental setup, including annotation conversion, module training, and the tools used in the execution module. we then evaluate lumos by: 1) comparing lumos with existing open-source llm agents and gptbased agents, 2) contrasting lumos against other agent training methods, 3) manifesting lumos generalizability on two unseen tasks involving new environments and actions, and 4) assessing lumos annotation quality. ## conclusion we introduce lumos, an open-source, generalizable language agent training framework. we propose two formulations, lumos-i and lumos-o, which promote collaboration among agent modules to solve complex tasks. for module training data, we use llms to transform reasoning steps in existing benchmarks into a unified format applicable within lumos framework. lumos outperforms a variety of open-source agents across the 9 datasets. it performs even better than gpt agents on qa and web tasks. lumos also exceeds potential agent training formulations and exhibits superior generalization on two unseen interactive tasks.