The Art of Tool Interface Design (2025)

Abstract: We present an agentic framework, Thinker, which achieves state of art performance in challenging reasoning tasks for realistic customer service scenarios that involve complex business logic and human interactions via long horizons. On the \tau -bench retail dataset, Thinker achieves 82.6% success rate with GPT-4o (version 2024-06-01) (baseline: 68.3%), and 81.9% success rate with Llama-3.1 405B (baseline: 49.6%), without any fine-tuning. Thinker effectively closes the gap in reasoning capabilities between the base models by introducing proper structure.The key features of the Thinker framework are: (1) State-Machine Augmented Generation (SMAG), which represents business logic as state machines and the LLM uses state machines as tools. (2) Delegation of tasks from the main reasoning loop to LLM-powered tools.(3) Adaptive context management.Our prompting-only solution achieves signficant gains, while still maintaining a simple and standard agentic architecture with a ReAct style reasoning loop. The key is to innovate on the tool interface design, as exemplified by SMAG and the LLM-powered tools.

## introduction recently, there has been increasing excitement around the potential of llm agents to enable new levels of automation across various industries. however, the deployment of llms in real world systems is not at all trivial. recently, sierra's research team published τbench @xcite , a challenging public dataset to evaluate ai agents' performance and reliability for real-world customer support scenarios. τ -bench allows automatic end-to-end evaluation by having a user model talking to an ai agent, where the user model is an llm instructed by a script of task instructions (see appendix c for examples). as sierra ceo bret taylor noted @xcite , results of "agents built with simple llm constructs (like function calling or react)" still result in poor performance, indicating an urgent need for more structured and effective agent architectures. it is clear that we need more robust agent architectures that can handle business logic reliably and talk and adapt flexibly like a human. we present an agentic framework with the following features: the contributions of this paper are: ## the thinker framework an illustration of thinker is given in figure 1 . there is a main reasoning loop that handles each turn, where in each step, the llm call generates a response which could either involve some tool calls or responding to the user. if the llm outputs tool calls, the tool calling results are then added to the working memory for subsequent reasoning. allowing multiple tool calls in one round not only im-proves efficiency but also more effectively supports requests that involve multiple tasks (e.g., change address for all my pending orders). this is natively supported in gpt4 via the functional calling support; we have implemented the equivalent functionality for llama-3 via prompting and postprocessing. the loop exits when the llm decides to respond to the user. to improve reasoning, the response starts with a "thought" element. since the main reasoning loop involves react-style thoughts and gpt-style multi-function calling, the high level orchestration structure is essentially react @xcite and gpt-style multi-function calling (openai, 2023) combined. as illustrated by figure 1 , there are three types of tool calls: (1) simple tools that correspond to the raw apis (2) llm-powered intelligent tools which takes natural language inputs and may involve an internal multi-step workflow to carry out a specific sub-task (section 2.2) (3) flows (state machines), which model business processes and rules that involve multi-turn user interactions (section 2.1). the tool call results, chat history, system instructions, available tools, and active flows collectively form the working memory. thinker uses adaptive context management to optimize the context before feeding into the llm (section 2.3). ## walk through via an example we use an example to walk through how thinker solves a typical customer service case. figure 2 presents an example execution log with llama-3 on the τ -bench retail dataset. we have added illustrations of each decision round. user: hi! i recently won a lottery and would like to upgrade all my items to the most expensive options available. please ensure that the shoe size remains the same. can you assist me with this? [user not authenticated. before taking any actions, you have to first authenticate the user identity by locating their user id via email, or via name + zip code. this has to be done even when the user already provides the user id.] note that the annotation was a state-dependent instruction from smag. after authentication and checking the user's orders, there are a few rounds of clarification where the assistant identified 3 orders but only one of them is pending. then the user requested to upgrade the order. figure 2 then illustrates the steps starting from this turn. thinker executes a reasoning loop to perform multiple rounds of reasoning and tool calls to arrive at a response. in each round, it generates an output structure that has a thought part, which performs chain-of-thought reasoning, followed by an action part, which could either be a list of actions or a response to the user. in round 1, the agent outputs 4 calls to find_product_items. we have also prepared an llm-powered intelligent tool, find_product_items, which finds the matching product items based on a natural language description. notice to comply with the nuanced requirement "ensure the shoe size remains the same", the agent generated a requirement argument of "most expensive, size 9". the llm-powered tool then loads all available shoe items. the llm call in the tool translates the requirement into a filter condition "size": ["9"] and a post-processing filter "most expensive" -this is essentially mapping the natural language requirement "most expensive, size 9" to sql statement "select * from products where @xmath0. then executing such filtering logic yields a single product item. the tool call results generate additional messages and are appended to the execution trace. we noticed that even gpt-4o still makes mistakes at direct "data processing" tasks but casting the problem as a "sql-like query generation" followed by deterministic data processing is a lot more reliable. then in round 2, thinker used smag to enforce the business logic that a mutation action must be first presented to the user for confirmation. since we have not asked the user for confirmation, the state machine logic thus returns a templated confirmation message and some state-dependent instructions to instruct the llm to ask for confirmation. another nuance is that if the user needs an address change to this pending order, the address change must be finished before the item modification. thus the templated confirmation suggestion includes a sentence that asks whether the user needs an address change, to comply with the business logic. given the templated confirmation suggestion and the llm instructions for this state, in the next round, the llm summarized the templated confirmation suggestion into a response message. since this round has no more tool calls, this exits the reasoning loop and the response is sent to the user, asking whether the user needs an address change and whether the user confirms the change. in the next turn, the user then confirms the change and replies that no address change is needed. the agent then carries out the actual mutation logic, since the agent parsed a state-progressing user input (@xmath1. finally, in the next turn the llm generated a response with the modification details to the user. this successfully fulfilled the user's needs. ## experiments we conducted our experiments on the τ -bench retail dataset with gpt-4o (through azure api, @xmath2.1 405b model, with @xmath3. the τ -bench framework allows automatic end-to-end evaluation by having a user model talking to an ai agent, where the user model is an llm instructed by a small script (task instructions, see appendix c for some examples). we noticed issues in some task instructions (errors and ambiguities that confuse the user model), so we made some minor changes (listed in appendix a); these changes are applied to the baselines as well. for the user model, we use the gpt-4o model. we further optimized the user model via a chain-of-thought prompt (appendix b.1) so that it starts generation by recalling the relevant instructions, then recalls the unfulfilled instructions (to better keep track of the remaining things to do), and finally gives a response. this lead to a 1.8% success rate improvement on llama-3. we consider cot user model results as the baseline since our focus is on ai agent performance. fixing the task specifications and improving the user model minimizes user side errors and makes it possible to solve challenging tasks (if the user model makes a mistake, no agent can fix it. such user errors render the task less interesting). in all experiments, we collect 5 runs and report the average accuracy. in thinker agent implementations with both gpt-4o and llama-3, the llmpowered tools are all based on llama-3. ## conclusion we propose the thinker agent framework with the following key techniques: on the τ -bench retail dataset, these techniques come together to result in state-of-the-art performance. note that the large gain is achieved via a prompting-only solution and it is using a standard agentic architecture with a react style reasoning loop. we highlight the power of careful tool interface design, as exemplified via smag and llmpowered tools. we believe this is a valuable insight regarding future agentic architectures.