NAT : Enhancing Agent Tuning with Negative Samples (2025)

Abstract: Interaction trajectories between agents and environments have proven effective in tuning LLMs into task-specific agents. However, constructing these trajectories, especially successful trajectories, is often computationally and time intensive due to the relatively low success rates of even the most advanced LLMs, such as GPT-4 and Claude. Additionally, common training paradigms like supervised fine-tuning (SFT) and reinforcement learning (RL) not only require large volumes of data but also have specific demands regarding the trajectories used. For instance, existing SFT approaches typically utilize only positive examples, limiting their efficiency in low-resource scenarios. To address this, we introduce Negative-Aware Training (NAT), a straightforward yet effective method that leverages both successful and failed trajectories for fine-tuning, maximizing the utility of limited resources. Experimental results demonstrate that NAT consistently surpasses existing methods, including SFT, DPO, and PPO, across various tasks.

## introduction an agent is a model that can interact with environments, make decisions, and achieve predefined goals @xcite . recent work has built powerful llms such as @xcite , prompting them as the core of an agent system to process information and make decisions with few-shot examples @xcite . however, these agents depend on closedsource, paid apis, raising concerns about cost and latency, and face limitations as llms are not specifically designed for tasks like action generation or tool use, with few-shot prompting offering only limited learning support @xcite . tuning llms on interaction trajectories between agents and environments has proven effective in transforming them into task-specific agents @xcite @xcite @xcite . however, building these trajectories, especially successful ones, is resource-intensive and time-consuming. even advanced models like @xcite and claude @xcite , struggle with tasks requiring complex planning, reasoning, or tool use, resulting in low success rates. this scarcity of successful trajectories turns agent-tuning into a lowresource scenario, where the lack of high-quality data becomes a significant bottleneck. traditional approaches to fine-tuning, such as supervised fine-tuning (sft) and reinforcement learning (rl), heavily depend on a large volume of successful trajectories to be effective. supervised fine-tuning, in particular, tends to focus on positive examples-those where the task is completed successfully-discarding failed trajectories, or negative examples, as irrelevant or uninformative. reinforcement learning-based methods like proximal policy optimization (ppo) attempt to address this issue by exploring a broader range of trajectories, but they still face challenges when dealing with complex, low-success-rate tasks @xcite . our preliminary observations show that over 80% of data generated during typical interaction-based data collection is discarded due to failure, leading to resource wastage and undertrained agents despite multiple rounds of data generation. to address these inefficiencies, we propose a novel approach to fine-tuning agents that leverages both successful and failed trajectories more effectively. our method, called negative-aware training (nat), builds on the insight that negative examples contain valuable information that can complement positive examples when used appropriately. rather than discarding negative data, nat uses both positive and negative trajectories during fine-tuning, providing the model with richer information about what works and what doesn't in a given task. through simple but effective modifications, nat enables llm-based agents to better generalize from limited resources by learning from a more diverse range of examples. our experimental results demonstrate that nat significantly outperforms traditional methods, including sft, dpo, and ppo, across a range of tasks. nat is particularly effective in low-resource scenarios, where it delivers substantial improvements in mathematical reasoning and multi-hop question answering tasks, with gains of 8.74 points and 6.05 points, respectively, over traditional finetuning approaches. these results confirm our hypothesis that failed trajectories are valuable for agent tuning, and show that our approach, nat, is an effective and efficient way of utilizing failed trajectories. our contributions can be summarized as follows: ## analysis tables 3 and 4 showcase the capability of llms to learn from negative examples and the superiority of nat compared to other methods in low-resource scenarios. in this section, we delve into various factors that could influence the effectiveness of nat. specifically, we seek to address the following questions: since only the math task contains enough data for our experiments, the analysis is done on the math task. ## conclusion we introduced nat (negative-aware training), a simple yet effective approach that incorporates negative trajectories into the agent fine-tuning process. our experiments on mathematical reasoning and question-answering tasks highlight the superior performance of nat over existing methods for agent tuning, particularly in low-resource scenarios. we also conducted extensive analyses to uncover the factors contributing to the success of nat, providing insights into its effectiveness.