Agent- FLAN : Designing Data and Methods of Effective Agent Tuning for Large Language Models (2024)

Abstract: Open-sourced Large Language Models (LLMs) have achieved great success in various NLP tasks, however, they are still far inferior to API-based models when acting as agents. How to integrate agent ability into general LLMs becomes a crucial and urgent problem.This paper first delivers three key observations: (1) the current agent training corpus is entangled with both formats following and agent reasoning, which significantly shifts from the distribution of its pre-training data; (2) LLMs exhibit different learning speeds on the capabilities required by agent tasks; and (3) current approaches have side-effects when improving agent abilities by introducing hallucinations. Based on the above findings, we propose Agent-FLAN to effectively Fine-tune LANguage models for Agents.Through careful decomposition and redesign of the training corpus, Agent-FLAN enables Llama2-7B to outperform prior best works by 3.5% across various agent evaluation datasets. With comprehensively constructed negative samples, Agent-FLAN greatly alleviates the hallucination issues based on our established evaluation benchmark. Besides, it consistently improves the agent capability of LLMs when scaling model sizes while slightly enhancing the general capability of LLMs. The code and models are available at https://github.com/InternLM/Agent-FLAN.

## introduction language agent @xcite , which leverages the powerful abilities of llms to perceive environments, make decisions, and take actions, has emerged as an effective solution to complicated real-world problems @xcite @xcite . existing studies mainly focus on either prompt engineering (wei et al., 2022; yao et al., 2022b; shinn * corresponding author @xcite or framework scheduling of one/multiple close-sourced llm (gpt-4) @xcite @xcite to accomplish agent tasks. despite the impressive results and flexibility of these works, close-sourced llms suffer from steep financial cost and safety issues @xcite , impeding them from further deployment. recently, open-sourced llms emerged as effective alternatives and show promising results in various applications @xcite @xcite . however, a lingering challenge persists: while these llms excel in tasks necessitating linguistic finesse, they fall short when deployed as agents, especially when compared to api-based models @xcite . efforts have been made to finetune llms for specific vertical fields @xcite , but this approach risks compromising their general capabilities. thus, integrating effective agent abilities into general llms becomes a critical and pressing challenge in the field, aiming to bridge the gap between specialized and versatile language models. in this paper, we embark on a journey to mitigate this disparity between the prowess of open-sourced llms and the superior performance exhibited by api-based models in agent-centric tasks. we first unravel three critical observations that serve as the foundation for our exploration: (1) most agent training data is entangled with both format following and general reasoning, deviating from the original pretrain domain of the model, i.e., natural conversation, leading to inadequate agent learning, (2) by explicitly decomposing the training data along the basic capabilities, each loss exhibits different convergence curves, indicating varied learning speeds on the abilities requisite for agent tasks, and (3) the existing approaches that predominantly concentrate on specific agent abilities, overlooking the prevalence and significance of hallucinations in the model's output. building upon these key insights, we present agent-flan, which investigates the designing choice of effective agent finetuning for large language models. specifically, we tease apart format following and common reasoning in the agent training corpus, aligning the fine-tuning process to the pretrained domain of the language model. this elicits the pure agent abilities in llms without overfitting to specific format protocols. by further disentangling the agent tasks into distinct facets along the fundamental competencies of llms, agent-flan affords training flexibilities depending on the varying learning rates of respective competencies. to comprehensively resolve the hallucination issues in agent tasks, we construct the agent-h benchmark, which assesses the hallucination issues of llms from various aspects. subsequently, we meticulously curate diverse 'negative' training samples to mitigate this problem effectively. we employ agent-flan on the open-sourced llama2-series, which surpasses prior works by a substantial 3.5% margin across a spectrum of agent evaluation benchmarks, including general agent tasks and tool utilization. furthermore, we provide a deeper understanding of the dynamics involved in agent tuning: the scaling laws governing data and model dimensions, and the intricate relationships between general and agent-specific tasks. our major contributions are as follows: ## pilot observations in this section, we delve into three pivotal observations on agent tuning that serve as the foundation of our subsequent investigation. observation 1. most agent training data is entangled with both format following and general reasoning, causing a significant departure from the model's original pretraining language domain, namely, natural conversation. recent agent tuning works @xcite endorse the adoption of specific formats, exemplified by react (thought-action-observation), for fine-tuning the language model. moreover, it's noteworthy that action arguments are frequently presented in json format. encoding both format and reasoning knowledge into the training corpus shifts the tuning process from the original chat domain, presenting it as an out-ofdomain task for language models. as shown in figure 2 , we compare the training curve of formatted data and normal data. it can be clearly seen that the loss associated with formatted data descends more rapidly to a low value, while keeping content loss still high (0.54 vs 0.04), indicating that the former leads to an inadequate learning process. this phenomenon can be probably attributed to the existence of a fixed structure (react, json), where the model quickly gets overfitted to the format itself. consequently, it fails to grasp the underlying reasoning abilities embedded within the training data, resulting in unsatisfied performance. observation 2. by explicitly decomposing the training data along the basic capability aspects, each loss exhibits different convergence curves, indicating varied learning speeds on the capabilities requisite for agent tasks of llms. inspired by @xcite , we explicitly disentangle the model's capabilities into distinct components: instruction following, reasoning, retrieval, and understanding. in this context, instruction following corresponds to format generation, reasoning corresponds to the thought quality at each step, retrieval involves selecting the appropriate function name to execute the task, and the understanding encompasses the parameter inputs for the selected functions. by visualizing the loss based on respective aspects in figure 3 , we discern that llm tends to exhibit varying learning speeds for the capabilities essential for proficient agents. to elaborate, retrieval and understanding emerge as relatively more manageable tasks compared to reasoning, with instruction following being the simplest in the learning process. this observation serves as a compelling motivation to further disentangle the training data along these model capabilities and subsequently balance these data based on the model's varying learning rates. observation 3. existing approaches predominantly concentrate on specialized agent abilities, overlooking the prevalence and significance of hallucinations in the model's output. agenttuning @xcite introduces mixture training, where both a general dataset and an agent dataset are simultaneously provided to the model during the tuning process. while this strategy indeed leads to steady performance improvements, we observe that it has limited impact on addressing hallucination issues, i.e., a crucial concern often overlooked in recent agent studies. this concern becomes particularly significant when deploying language models in real-world applications, as illustrated in figure 4 . hallucination manifests in two primary aspects: (1) when the model is required to invoke a response, it strictly adheres to the training format, disregarding the user-generated query, and (2) the model is susceptible to being triggered with non-existent functions when presented with induced questions. this underscores the necessity of directing more attention toward refining agent tuning mechanisms and the establishment of appropriate benchmarks to assess and mitigate agent hallucination effectively. ## agent-flan recent works start to explore the effectiveness of finetuning language models on various agent tasks, with varied data quality, model sizes, and tuning approaches. building on the pilot observations, we dive deeper into the data and method designing choice of effective agent tuning and discuss three key improvements to the language agent models. ## conclusion in this work, we investigate the design choice of data and methods in agent tuning for llms. by first carrying out three critical observations, we pinpoint the current bottleneck in bridging the gap between the open-sourced llms and api-based models. based on this, we present agent-flan to effectively fine-tune the language models for agent tasks. through careful decomposition and redesign of the existing training corpus, agent-flan em-powers llama2-7b to outperform previous works significantly on a wide spectrum of agent tasks.