Enhancing the General Agent Capabilities of Low-Paramter LLM s through Tuning and Multi-Branch Reasoning (2024)

Abstract: Open-source pre-trained Large Language Models (LLMs) exhibit strong language understanding and generation capabilities, making them highly successful in a variety of tasks. However, when used as agents for dealing with complex problems in the real world, their performance is far inferior to large commercial models such as ChatGPT and GPT-4. As intelligent agents, LLMs need to have the capabilities of task planning, long-term memory, and the ability to leverage external tools to achieve satisfactory performance. Various methods have been proposed to enhance the agent capabilities of LLMs. On the one hand, methods involve constructing agent-specific data and fine-tuning the models. On the other hand, some methods focus on designing prompts that effectively activate the reasoning abilities of the LLMs. We explore both strategies on the 7B and 13B models. We propose a comprehensive method for constructing agent-specific data using GPT-4. Through supervised fine-tuning with constructed data, we find that for these models with a relatively small number of parameters, supervised fine-tuning can significantly reduce hallucination outputs and formatting errors in agent tasks. Furthermore, techniques such as multi-path reasoning and task decomposition can effectively decrease problem complexity and enhance the performance of LLMs as agents. We evaluate our method on five agent tasks of AgentBench and achieve satisfactory results.

## introduction large language models (llms) have been extensively employed in a wide range of natural language processing tasks, yielding groundbreaking achievements. furthermore, llms have demonstrated their capability to undertake more challenging tasks, such as functioning as ai agents. unlike conventional reasoning tasks, an ai agent is an entity that needs to interact with the human or external environment, draw inferences, and judge subsequent actions based on feedback. each single task typically involves multiple rounds of dialogue to accomplish. for instance, in a home environment, an agent may be tasked with various household tasks that require continuous interaction with the environment. the agent needs to evaluate its actions based on the feedback from the environment and make timely adjustments to its strategies. traditional ai agents are usually effective in specific domains or environments, but their generalization and adaptability are obviously insufficient @xcite . in recent years, an increasing number of work @xcite @xcite @xcite have demonstrated that llms possess strong capabilities in reasoning, planning, memory, and utilizing external tools. this has propelled llms towards becoming more generalized and adaptive agents. recently, agentbench @xcite conducts extensive evaluations of both commercial and opensource llms on eight different agent tasks. the results reveal that commercial api models show superior agent capabilities. in addition, work such as autogpt @xcite and gpt-engineer @xcite also use llms as agents to build a complete framework for solving complex real-world problems. however, open-source models, especially those with smaller parameter sizes, still have substantial potential for enhancement. as shown in fig. 1 , the average performance of 7b and 13b llms on each agent task is significantly lower than the commercial models. unlike commercial llms, small-scale opensource llms are relatively inefficient in general knowledge @xcite . besides, lower parameter sizes limit reasoning and memory capacity, often leading to hallucinations in the agent dialogue process @xcite . however, in practical applications, llms with 7b and 13b parameters are the most widely used due to their relative ease of deployment and fine-tuning. therefore, enhancing the capabilities of such llms is of great practical significance. currently, studies on llms agents or enhancing model reasoning capabilities @xcite primarily focus on large-scale models. the investigation of agent capabilities on 7b and 13b llms is still in its early stages of exploration. as explained, a proficient agent requires task-planning abilities, proficiency in utilizing external tools, and long-term memory capabilities. task planning refers to the ability of the model to decompose large-scale tasks into manageable sub-goals, facilitating efficient handling of complex tasks. long-term memory capabilities reflect the ability of the llms to retain and recall historical information during their interactive processes with the environment. considering these abilities, we propose a method to enhance the performance of 7b and 13b llms on agent tasks. in our proposed approach, we focus on enhancing the agent capabilities of llms from two key aspects. first, improving the agent capabilities through supervised fine-tuning (sft). this approach fundamentally enhances the llms themselves. unlike general reasoning tasks, an agent's role goes beyond planning and reasoning. it also involves continuous interaction with the environment or humans to execute subsequent actions until a desired outcome is achieved. to improve the agent abilities of llms, it is essential to train them on diverse datasets that reflect the full range of interactive behaviors between the agent and the environment. this involves constructing data that not only records the actions taken by the agent but also captures the internal thought processes and decision-making. additionally, the environment should provide meaningful feedback to guide the learning of the agent. we propose to use @xcite to construct data. by designing a framework that involves gpt-4 engaging the multiturn dialogues, we can generate conversational data that captures the interaction between different roles. during these conversations, gpt-4 can take on different roles, such as playing the part of an agent, a user, or the environment, and actively participate in dynamic exchanges. in addition, we incorporate a significant amount of general instruction tuning data into the constructed dataset to preserve the general capabilities of the llms. besides, we optimize the reasoning path through task decomposition and backtracking. inspired by chain of thought @xcite , significant efforts have been dedicated to activating the reasoning ability of the llms. for instance, react @xcite integrates the thinking process into the task of multi-step reasoning. tot @xcite uses depth-first and breadth-first traversal of reasoning nodes, which is more conducive to finding the optimal solution. we migrate the idea of tot to the agent tasks and combine it with task decomposition and backtracking. task decomposition leverages the task planning capability of the llms to decompose complex and lengthy tasks into several smaller subtasks. considering that it is difficult for llms to find optimal answers or complete tasks through a single reasoning path, we introduce a judgment process where the reasoning process goes back to the starting point, termed backtracking. through the integration of task decomposition and backtracking, we aim to enhance llms' ability to handle complex tasks effectively. the main contributions of this paper are: 1) we explore the capabilities of 7b and 13b open-source llms as agents, exploring their potential in performing agent tasks. 2) we propose supervised fine-tuning with specific agent data as a fundamental approach to improving the capability of opensource llms as agents. to achieve this, we develop a method for constructing agent data. 3) we find that task decomposition and backtracking are effective approaches for addressing complex agent tasks. we conduct experiments on agentbench and achieve promising results. large language model as agent. with the rapid advancement of llms, extensive research has been conducted to explore their powerful capabilities in planning and reasoning @xcite . this has opened up the possibility of employing llms as agents. on the one hand, there have been several efforts to apply llms to various agent tasks and construct agent simulation frameworks. on the other hand, several works @xcite , such as react @xcite , have focused on incorporating reasoning and deliberation into the agent process for llms. in addition, some works apply the reasoning methods to the agent interaction process. pet @xcite applies task decomposition to the household agent environment, which is helpful for llms to complete complex tasks. lats @xcite and rap @xcite apply monte carlo tree search to the agent reasoning process. it is advantageous to find better answers compared with tot. in addition, research works such as autogpt @xcite and gpt-engineer @xcite utilize commercial llms as agent core of their frameworks, enabling the development of comprehensive agent architectures to tackle complex real-world problems. instruction tuning for language model. in-struction tuning plays a crucial role in training llms. after pre-training with massive unsupervised data, llms acquire a substantial amount of knowledge and process language understanding and generation capabilities. further supervised instruction fine-tuning @xcite is conducted to align the model with human instructions and generate outputs that better align with human preferences. instruction tuning mainly focuses on constructing complex and diverse general-purpose tasks to train llms to answer questions in a human manner. for example, flan @xcite and t0 @xcite construct a multi-task instruction tuning dataset using massive publicly available datasets. the fine-tuned model shows strong zero-shot generalizability. in addition to utilizing existing datasets, another common approach is to generate data using commercial llms. self-instruct @xcite leverages gpt-4 to generate a large amount of diverse data, given a few seed tasks. these data are used for fine-tuning open-source llms and get significant improvements in various tasks. to enhance the agent capability of llms, agenttuning @xcite utilizes commercial llms to construct data in specific agent environments containing multi-turn dialogues. ## methodology in this section, we first give a formal definition of llms as agents. then, we introduce the two components of our approach. in the first part, we construct agent-tuning data to fine-tune llms with parameter-efficient tuning methods. this is a way to fundamentally improve the capabilities of llms. in the second part, we propose enhancing the reasoning capabilities of llms through task decomposition and backtracking. ## conclusion llms as intelligent agents have demonstrated powerful agent capabilities. in this work, we explore the 7b and 13b llms as agents, and propose to enhance the agent performance of these open-source models by supervised fine-tuning through agent data as well as multi-branch reasoning. sft can effectively reduce format errors and hallucination output of the llms, which not only improves the agent performance but also facilitates the application of various reasoning methods to agent tasks.