A gent B ank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories (2024)

Abstract: Fine-tuning on agent-environment interaction trajectory data holds significant promise for surfacing generalized agent capabilities in open-source large language models (LLMs). In this work, we introduce AgentBank, by far the largest trajectory tuning data collection featuring more than 50k diverse high-quality interaction trajectories which comprises 16 tasks covering five distinct agent skill dimensions. Leveraging a novel annotation pipeline, we are able to scale the annotated trajectories and generate a trajectory dataset with minimized difficulty bias. Furthermore, we fine-tune LLMs on AgentBank to get a series of agent models, Samoyed. Our comparative experiments demonstrate the effectiveness of scaling the interaction trajectory data to acquire generalized agent capabilities. Additional studies also reveal some key observations regarding trajectory tuning and agent skill generalization.

## introduction an agent is an entity that possesses the capability for volition, decision-making, action-taking , and, most critically, environment perception @xcite . in the realm of cognitive science, previous literature has suggested that interaction with environment derives an agent's generalized intelligence, and intelligent behavior emerges from a synergistic blend of simpler behaviors, including reasoning, programming, and game playing @xcite . the proprietary large language models (llms), such as gpt-3.5 @xcite and @xcite , have demonstrated strong capabilities in instruction following, reasoning, and planning, which encourage many attempts to build autonomous agent systems utilizing llms as core controllers @xcite . however, comprehensive evaluations have shown that the majority of open-sourced llms * corresponding authors. previous research pointed out that learning from gold interaction trajectories, a process we term trajectory tuning, could enhance the capabilities of weaker agents @xcite . early studies heavily focus on specialized agents designed for particular tasks. existing attempts are exemplified by @xcite and @xcite , who build agent trajectory data from teacher agents (e.g., gpt-4) and fine-tune opensource llms to improve specific agent abilities like reasoning. taking a step further, @xcite adopt a multi-task tuning approach called agenttuning. however, trained on a small trajectory dataset comprising six tasks with 1.8k trajectories, @xcite struggle to enhance the generalized agent capability, especially in the case of 7b and 13b models. to explore the impacts of incorporating interaction trajectory data on agent ability generalization, we construct agentbank, the largest agent interaction trajectory dataset to date. agentbank features 16 distinct tasks across five agent skill dimensions and contains over 50,000 trajectories, each annotated with high-quality chain-of-thought (cot) rationale for every step of action. leveraging a novel annotation pipeline that fully exploits the capability of llms, the trajectory collection process is highly scalable and adaptable to diverse agent environments. in contrast to prior studies that have relied on successful trajectories of gpts for training data @xcite , agentbank stands out with its exceptional quality and mitigated susceptibility to the difficulty bias issue. we further develop samoyed, a suite of models with enhanced agent capabilities, through the trajectory tuning of llama-2 @xcite using agentbank. our evaluations on both held-in and unseen held-out tasks suggest that by fine-tuning agentbank fireact agentinstruct agent-flan agentohana (this work) @xcite ) @xcite on extensive multi-task trajectories, our models exhibit remarkable agent intelligence in comparison with untuned ones. specifically, samoyed outperforms gpt-3.5-turbo on average on held-in tasks, which can be attributed to the in-domain trajectory tuning. furthermore, our models also demonstrate superior performance on held-out tasks, underscoring the efficacy of large-scale trajectory tuning in acquiring generalized agent capabilities. to trace the emergence of agent capabilities generalization, we follow the initial evaluation with a systematic analysis across various dimensions. initially, we delineate the scaling trends of tasks alongside the quantity of trajectories. next, we conduct an ablation study that merges generalist instruction data and code data to examine the benefits of hybrid training. this study uncovers further enhancements in the agent capabilities and mitigates catastrophic forgetting. furthermore, our findings underscore the pivotal role of cot rationale in the acquisition of generalized agent capability. our contributions are summarized as follows: ## train samoyed with agentbank to initialize the training of samoyed, we formulate agent interaction trajectories in agentbank into a chatbot-style schema (u, a 1 , o 1 ..., a i , o i , ..., a n ), where u is the task instruction, o i and a i denote the observation from the task environment and the corresponding action with rationale generated by the agent in the i-th round. during the training process, we feed the entire interaction trajectory into a decoder-only llm, where only the auto-regressive loss on tokens of ground-truth responses @xmath0, ..., a n } is counted. we mask all tokens belonging to the instruction and observations from the environment to prevent them from loss computation. concretely, task skill dim. #inst. metric held-in tasks hotpotqa (yang et al., 2018) reasoning 100 exact match strategyqa @xcite reasoning 100 exact match gsm8k @xcite math 100 exact match math @xcite math 100 exact match ic-sql @xcite programming 100 avg. reward mbpp @xcite programming 100 success rate mind2web @xcite al., 2023) web 1173 step sr webshop (yao et al., 2022a) web 200 avg. reward alfworld (shridhar et al., 2020b) embodied 134 success rate held-out tasks bamboogle (press et al., 2022) reasoning 126 exact match theoremqa (chen et al., 2023b) math 100 exact match ic-bash (yang et al., 2023) programming 200 avg. reward miniwob++ (kim et al., 2023) web 460 success rate scienceworld (wang et al., 2022a) embodied 270 avg. reward recent studies @xcite suggest that hybrid training with generalist instruction data and code data may improve the generalized ability of llm agents. following them, we adopt a mixture of agentbank d agent , the general domain instruction dataset d general , and the code dataset d code for fine-tuning. we perform detailed ablation experiments to explore the effectiveness of generalist and code data in section 7.2. training setup we use adamw optimizer with a learning rate of 5e-5 and a cosine scheduler. the models are trained for 3 epochs with 3% warm-up steps. the batch size is set to 128 and the sequence length is 2048. we choose sharegpt 1 as the generalist instruction data, and evol-codealpaca @xcite as the code data. the mixture ratio of d agent , d general , and d code is 80%, 10%, 10%. a corresponding data contamination analysis can be found in appendix e. all experiments are conducted on 8 nvidia a100 80g gpus. we use fastchat @xcite and pytorch fsdp @xcite for efficient training. held-in/out tasks in an effort to balance the reliability and efficiency of the evaluation, we select nine tasks from agentbank to form the held-in test set. for tasks with a huge test set, following @xcite , we randomly sample a subset from the original test set. to evaluate the generalized agent intelligence of samoyed, we additionally compile five unseen held-out tasks that do not exist in agentbank but still fall into the five skill dimensions of a foundation agent. the held-in and held-out evaluation tasks used in the experiments are listed in ## conclusion in this work, we explore the acquisition of generalized agent capabilities through fine-tuning opensource llms on massive interaction trajectories. we introduce by far the largest interaction trajectory dataset agentbank, comprising over 50k trajectories that encompass 16 tasks across five distinct agent skill dimensions. building upon agentbank, we fine-tune llama-2 to develop samoyed, an open-source llm series specialized for agent tasks. evaluations on both held-in and held-out tasks show that samoyed significantly outperforms strong baselines in terms of generalized agent capabilities. comprehensive analysis also reveals the effectiveness of data mixture and plots the scaling law of trajectories. we hope this work to serve as a catalyst for further exploration in the development of more powerful agents.