A uto A ct: Automatic Agent Learning from Scratch for QA via Self-Planning (2024)

Abstract: Language agents have achieved considerable performance on various complex question-answering tasks by planning with external tools. Despite the incessant exploration in this field, existing language agent systems still struggle with costly, non-reproducible data reliance and face the challenge of compelling a single model for multiple functions. To this end, we introduce AutoAct, an automatic agent learning framework for QA that does not rely on large-scale annotated data and synthetic planning trajectories from closed-source models (e.g., GPT-4). Given limited data with a tool library, AutoAct first automatically synthesizes planning trajectories without any assistance from humans or strong closed-source models. Then, AutoAct leverages a division-of-labor strategy to automatically differentiate based on the target task information and synthesized trajectories, producing a sub-agent group to complete the task. We conduct comprehensive experiments with different LLMs, which demonstrates that AutoAct yields better or parallel performance compared to various strong baselines. Further analysis demonstrates the effectiveness of the division-of-labor strategy, with the trajectory quality generated by AutoAct generally outperforming that of others.

## introduction language agents @xcite @xcite , which leverage the powerful reasoning capabilities @xcite of large language models (llms) to interact with executable tools, have emerged as essential components of ai systems designed to address complex question-answering tasks @xcite @xcite @xcite . the process of endowing llms with such interactive capabilities is referred to as agent learning wherein planning @xcite plays a pivotal role, which is responsible for decomposing complex questions into simpler ones @xcite @xcite , invoking external tools @xcite @xcite , reflecting on past mistakes @xcite , and aggregating information from various sources to reach the final answer. there have been a lot of works @xcite @xcite @xcite ) that directly prompt closed-source off-the-shelf llms to plan on particular tasks. despite their convenience and flexibility, closed-source llms inevitably suffer from unresolved issues, as their accessibility often comes at a steep price and their black-box nature makes the result reproduction difficult. in light of this, some recent endeavors have shifted their focus towards imbuing open-source models with planning capabilities through fine-tuning @xcite @xcite . however, despite the achievements of the existing fine-tuning-based methods, they are not without limitations. on the one hand, training opensource models necessitates a substantial amount of annotated qa data pairs and still relies on closedsource models to synthesize planning trajectories. however, fulfilling these requirements in many real-world scenarios, such as private personal bots or sensitive company business, often proves to be rocky. on the other hand, from the perspective of agent framework, fine-tuning-based methods compel one single language agent to learn all planning abilities, placing even greater pressure on them. these contradict simon's principle of bounded rationality @xcite , which states that "precise social division-of-labor and clear individual tasks can compensate for the limited ability of individuals to process and utilize information". to this end, we introduce autoact, an automatic agent learning framework for qa, which does not rely on large-scale annotated data and synthetic trajectories from closed-source models while incorporating explicit individual tasks with precise division-of-labor (see fig. 1 ). given a limited set of user-provided data examples, autoact starts with a meta-agent to obtain an augmented database through self-instruct @xcite . then, armed with a prepared tool library, the meta-agent can automatically synthesize planning trajectories without any assistance from humans or strong closed-source models. finally, we propose the division-of-labor strategy which resembles cell differentiation based on the self-synthesized trajectories (genes), where the meta-agent acts as a stem cell @xcite and differentiates into three sub-agents with distinct functions: task decomposition, tool invocation, and self-reflection, respectively. our differentiation process is essentially a parameter-efficient training process on the self-synthesized trajectories with low-consumption resources. we list the differences between au-toact and prior works in tab. 3 . experiments on complex question-answering tasks with different llms demonstrate that au-toact yields better or parallel performance compared to various strong baselines. extensive empirical analysis demonstrates the effectiveness of our appropriate division-of-labor strategy. ## autoact in autoact, the meta-agent can be initialized with any kind of open-source model. target task information. in this paper, we mainly focus on agent learning from scratch, which means the task information at hand is quite limited, primarily encompassing three aspects: task name m, task description p, task data examples c. concretely, p represents a detailed description of the task's characteristics. @xmath0, a i } |c| @xmath1, where |c| is very small which users can effortlessly provide (e.g., a few demonstrations). for a more in-depth view of task information, please refer to appx. e. note that the task information serves as the only user-provided knowledge of the task for autoact to conduct automatic agent learning. ## experimental setup tasks and metrics. we evaluate autoact on hotpotqa @xcite and scienceqa @xcite . hotpotqa is a multi-hop qa task challenging for rich background knowledge, the answer of which is usually a short entity or yes/no. following @xcite , we randomly select 300 dev questions divided into three levels for evaluation, with 100 questions in each level. for hot-potqa, the @xmath2, 1] is defined as the f1 score grading between the prediction and groundtruth answer. scienceqa is a multi-modal qa task spanning various scientific topics. we also divide the test set into three levels based on the grade, with 120 randomly sampled data in each level. since scienceqa is a multi-choice task, the @xmath3, 1} is exactly the accuracy. note that due to the limitations of lms in generating images, for scienceqa, during the self-instruct stage, we directly generate captions for the images instead. baselines. we choose the open-source llama-2 models @xcite and mistral-7b @xcite as the backbones of our meta-agent and sub-agents. the compared baselines include cot @xcite , react, chameleon @xcite , reflexion @xcite , bolaa @xcite , rewoo @xcite , fireact @xcite . we detail each baseline in appx. b. to ensure fairness, we maintain an equal training trajectory volume of 200 for fireact and autoact (200 synthesized data). as reflexion provides answer correctness labels during reflection but other methods including autoact do not, we test all the other methods twice and choose the correct one for evaluation. for all the prompt-based baselines, we uniformly provide two examples in the prompt. training setups. we fine-tune all our models with lora @xcite in the format proposed in alpaca @xcite . all the training and inference experiments are conducted on 8 v100 gpus within 16 hours. we detail the hyperparameters for training in appx. b. ## results compare to prompt-based agent learning baselines. as shown in tab. 1, the mistral-7b and llama-{13,70}b models consistently outperform various prompt-based baselines. the llama-70b model even surpasses the agent performance of gpt-3.5-turbo, achieving a rise of ↑ 3.77% on hotpotqa and ↑ 6.39% on scienceqa. therefore, whether in a single-agent or multi-agent architecture, prompt-based methods relying on few-shot demonstrations fail to precisely customize the behavior of the agent, which is also supported by the fact that fireact widely outperforms react and bolaa in the context of iterative planning. compare to fine-tuning-based agent learning baselines. further focusing on fireact in tab. 1, despite the aid of gpt-4, fireact's approach of assigning the entire planning task to a single model proves to be burdensome. as a result, its performance on scienceqa even falls short compared to the prompt-based global planning method, chameleon. autoact decouples the planning process and reaches a clear divisionof-labor among sub-agents for group planning, resulting in an improvement than fireact, with ↑ 5.77% on hotpotqa and ↑ 6.67% on scienceqa with llama-70b model. additionally, autoact achieves self-planning without relying on closedsource models and large-scale labeled datasets, which paves the way for automatic agent learning with open-source models from scratch. in ablation study ( §4) and human evaluation ( §5), we will further validate that the quality of trajectories synthesized by autoact is not inferior to fireact trained on trajectories synthesized using gpt-4. single-agent learning vs. multi-agent learning. under identical settings, multi-agent architectures generally exhibit better performance than single-agent (react vs. bolaa, fireact vs. autoact), which aligns with simon's theory of bounded rationality. seemingly contrary to expec- hotpotqa scienceqa autoact 48.47 78.61 -reflection 45.66 ↓2.81 75.28 ↓3.33 -multi 42.81 ↓5.66 69.72 ↓8.89 -fine-tuning 32.84 ↓15.63 61.94 ↓16.67 -filtering 32.51 ↓15.96 59.17 ↓19.44 easy medium hard overall 0 10 20 30 40 llama-2-7b-chat f1 score one three tool-specified easy medium hard overall 0 10 20 30 40 50 llama-2-13b-chat f1 score one three tool-specified easy medium hard overall 0 20 40 60 llama-2-70b-chat f1 score one three tool-specified figure 4: performance of autoact on hotpotqa based on different degrees of labor division. one is training a single model with all the differentiated data. three represents the differentiation into three agents: plan, tool, and reflect. tool specified indicates further differentiating the tool-agent with one tool, one agent. approach ablations. tab. 2 presents the performance of autoact on the llama-70b model after removing certain key processes. it can be observed that the least impactful removal is the -reflect. we investigate that in the zero-shot scenario, the model tends to be over-confident in its answers (as also confirmed in @xcite ). it typically only recognizes its errors when there are obvious formatting mistakes or significant repetitions in the planning process. consistent with previous findings, the removal of the -multi agents leads to a noticeable decrease in performance. a more exciting discovery is that the results of -multi are comparable to those of fireact. this indirectly suggests that the trajectory quality generated by the 70b model may be no worse than that of gpt-4. as expected, the performance deteriorates afterfine-tuning, which once again confirms the previous conclusion. to demonstrate the necessity of filtering out planning error data, we specifically remove the filtering process (-filtering) to examine the performance of autoact. the results indicate 0 20 40 60 80 100 overall action para action type thought rounds win rate (%) react bolaa fireact autoact 58 16 16 10 26 25 23 26 34 22 21 23 30 22 24 24 32 25 20 23 figure 6: human evaluation of trajectories generated by llama-2-70b-chat on hotpotqa. we compare the number of planning rounds, the logical correctness of thoughts, action types, action parameters, and the overall coherence of each trajectory. the figure above displays the win rate of each method in each aspect. ## analysis larger training data scale does not necessarily mean better results. we evaluate the influence of different training data scales on the performance of self-planning with llama-{7,13,70}b models on hotpotqa in fig. 3 (a-c ). it can be observed that the overall performance of different models goes to stability with minimal waves once the data scale exceeds 200. we speculate that this may be due to the limited ability of naive self-instruct to boost internal knowledge of the language model. as the training data increases, the knowledge which can be extracted through self-instruct decreases. despite our efforts to filter out duplicate data, the mindless increase can inevitably lead to a significant surge in similar data, which undermines the benefits of increasing the data scale and makes it challenging to improve model performance or even leads to over-fitting. to further confirm the role of training data, we decouple the models from the training data and evaluate their training results on trajectories synthesized by stronger models. from fig. 3 (d-f ), we can see consistent conclusions with previous findings. therefore, maximizing the diversity of the synthesized data in the database may be a key improvement direction for autoact and we leave this for our future work. we can also observe from fig. 3 (d-e ) that the larger the model, the higher the quality of the synthesized data, as the performance of the 7b model shows a gradual increase on self, 13b, and 70b synthesized data. ## conclusion and future work in this paper, we propose autoact, an automatic agent learning framework for qa that does not rely on large-scale annotated data and synthetic trajectories from closed-source models, while alleviating the pressure on individual agents by explicitly dividing the workload. interesting future directions include: i) expanding autoact to more realistic task scenarios @xcite @xcite , ii) boosting more knowledge via self-instruct (as analyzed in §5), iii) iteratively enhancing synthetic trajectories via self-improvement @xcite .