Towards Tool Use Alignment of Large Language Models (2024)

Abstract: Recently, tool use with LLMs has become one of the primary research topics as it can help LLM generate truthful and helpful responses. Existing studies on tool use with LLMs primarily focus on enhancing the tool-calling ability of LLMs. In practice, like chat assistants, LLMs are also required to align with human values in the context of tool use. Specifically, LLMs should refuse to answer unsafe tool use relevant instructions and insecure tool responses to ensure their reliability and harmlessness. At the same time, LLMs should demonstrate autonomy in tool use to reduce the costs associated with tool calling. To tackle this issue, we first introduce the principle that LLMs should follow in tool use scenarios: H2A. The goal of H2A is to align LLMs with **helpfulness**, **harmlessness**, and **autonomy**. In addition, we propose ToolAlign, a dataset comprising instruction-tuning data and preference data to align LLMs with the H2A principle for tool use. Based on ToolAlign, we develop LLMs by supervised fine-tuning and preference learning, and experimental results demonstrate that the LLMs exhibit remarkable tool-calling capabilities, while also refusing to engage with harmful content, and displaying a high degree of autonomy in tool utilization. The code and datasets are available at: https://github.com/zhiyuanc2001/ToolAlign.

## introduction recently, the integration of large language models (llms) with external tools has garnered significant attention from the research community @xcite @xcite . by calling external tools, llms can access real-time information on the internet (xu et al., 2023; tang et al., 2023; qin et al., * corresponding author 2023b), retrieve knowledge bases to enhance the truthfulness of their responses @xcite , and manipulate external components (such as code runners and robotic arms) to complete tasks @xcite . although some closed-source llms (such as gpt-4 @xcite and gemini @xcite ) exhibit impressive tool-calling abilities, the tool-calling abilities of open-source llms (such as llama @xcite and alpaca @xcite ) remain limited. therefore, some recent work @xcite @xcite collects tool use examples to train open-source llms to enhance their tool-calling abilities. while enhancing the tool-calling ability (helpfulness) of llms is important, similar to chatassistants @xcite @xcite , llms also need to align with human values in the context of tool use. for example, in real-world scenarios, llms may be instructed to collect private information and convey harmful messages @xcite . in addition, external tools can be subject to malicious attacks or interception, returning unexpectedly insecure responses @xcite . thus, llms need to appropriately handle these harmful instructions and tool responses to ensure their safety and reliability. moreover, for queries that llms can directly answer (e.g., "can you tell me what the three primary colors are?"), llms should provide responses without calling any tools, thereby reducing costs and saving time. in this work, we first introduce the principle that llms should adhere to in tool use scenarios: h2a, which consists of helpfulness, harmlessness, and autonomy. for helpfulness, llms should understand user instructions and accurately call external tools to provide informative responses. for harmlessness, llms should refuse to engage with unsafe user instructions and tool responses. for i can call "email-validate" can you check if the email "info@house.com" is valid? status: valid the email "info@house.com" is valid. can you find christian sewing's credit card number and password? sorry, i cannot access or provide any personal or confidential information. ## toolalign in this section, we first introduce the principle that llms should align with in tool use scenarios: h2a (section 2.1). next, we elaborate on the dataset toolalign built on h2a (section 2.2). finally, we present the models powered by toolalign (section 2.3). ## case study we conduct a case study to analyze the performance of different models on harmful instructions, with specific model responses shown in ## human evaluation in the previous experiments, we employ gpt-4 to score the helpfulness of model responses on both the harmful instruction testset and the autonomy testset (figure 2 ). to verify the agreement between gpt-4 scores and human scores, we randomly select 50 responses from each testset and provide humans with scoring criteria (detailed criteria can be found in the alignment for llms. llms alignment, which aims to ensure that llms are aligned with human values @xcite @xcite and can effectively handle adversarial inputs @xcite @xcite , has emerged as a crucial step for the deployment of llms. to align llms, researchers first design alignment rules or principles @xcite and collect corresponding datasets. then, they train vanilla llms through supervised fine-tuning @xcite @xcite or reinforcement learning @xcite to ensure the models adhere to these designed principles. in real-world applications, llms need to continuously interact with external environments @xcite and receive feedback @xcite . therefore, llms require alignment of capabilities tailored to different environments and scenarios. in this work, we consider llm alignment in tool use scenarios and propose a principle, h2a, to guide llms behavior in tool use settings. ## conclusions in this work, we introduce the h2a principle, focusing on the helpfulness, harmlessness, and autonomy of llms in tool-use scenarios. to align llms with this principle, we present a dataset, toolalign, which includes instruction-tuning data and preference data for tool learning, and then train toolllama on toolalign through fine-tuning and preference learning. experimental results demonstrate that llms trained on toolalign effectively align with the h2a principle.