Learning API Functionality from In-Context Demonstrations for Tool-based Agents (2025)

Abstract: Digital tool-based agents, powered by Large Language Models (LLMs), that invoke external Application Programming Interfaces (APIs) often rely on documentation to understand API functionality. However, such documentation is frequently missing, outdated, privatized, or inconsistent—hindering the development of reliable, general-purpose agents. In this work, we propose a new research direction: learning of API functionality directly from in-context demonstrations. This task is a new paradigm applicable in scenarios without documentation. Using API benchmarks, we collect demonstrations from both expert agents and from self-exploration. To understand what information demonstrations must convey for successful task completion, we extensively study how the number of demonstrations and the use of LLM-generated summaries and evaluations affect the task success rate of the API-based agent. Our experiments across 3 datasets and 6 models show that learning functionality from in-context demonstrations remains a non-trivial challenge, even for state-of-the-art LLMs. We find that providing explicit function calls and natural language critiques significantly improves the agent’s task success rate due to more accurate parameter filling. We analyze failure modes, identify sources of error, and highlight key open challenges for future work in documentation-free, self-improving, API-based agents.

## introduction in the past few years, ai agents have been rapidly adopted in society due to the development of large language models (llms) @xcite , allowing for superb performance in natural language understanding, summarization, and generation. more recently, toolbased agents have been introduced to increase the abilities of agents for specialized tasks that may even require up-to-date knowledge of external figure 1 : expert demonstration of the email.reply_email function extracted from workbench dataset @xcite . demonstrations are the basis of how agents understand functionality without prior documentation. a common, crucial requirement for api-based agents is access to api documentation that explains to the agent in natural language the functionality of the specific api functions available @xcite . the documentation explains both 1) what the function does and returns (if anything), and 2) what the input parameter schema is. both these components are important for tool selection and their execution by the agent. however, api documentation can frequently be unavailable, inconsistent, unstructured, or out-of-date @xcite @xcite . prior work in software engineering research on api documentation generation @xcite lacks evaluation of the generated documents via downstream tasks by an llm-based agent. other work @xcite generates api documentation by inputting a brief description of the function into an llm, and this documentation is then used by an agent. however, a rigorous analysis of learning functionality with no prior documentation or description remains absent in the api-based agent literature. we address these gaps by introducing and formalizing the problem of learning api functionality from scratch via in-context demonstrations of tool calls. demonstrations of tool calls are present within codebases of public repositories and logs. while prior work has finetuned agents on demonstrations @xcite , we focus on in-context learning to remove the need to update parameters. in this work, we analyze the ability of an api-based agent with a frozen llm with only access to demonstrations of tool calls to learn functionality to perform tasks. using various api datasets, we extract and standardize demonstrations for the agent to learn from, to perform the dataset tasks. our experiments revolve around three main pillars. we investigate the effect of 1) the number of demonstrations and 2) the representation of the demonstrations. specifically, we compare, in terms of downstream task performance, between providing the demonstrations directly, generating documentation from the demonstrations, or a combination of both. furthermore, we also show the impact of 3) collecting, evaluating, and summarizing experiences of the agent to update its understanding of the apis. our results consistently show across experiments the difficulties of the agent to perform tasks, highlighting how non-trivial learning of api functionality from in-context demonstrations is, even with state-of-the-art (sota) llms. we summarize our main contributions below: from our experiments, we find that a central recurrent problem is filling in parameter values, leading up to a 39% decrease in an agent's success rate if the method for processing demonstrations incorrectly describes the parameter schema. ## preliminaries: task completion with api-based agents consider a goal-conditioned partially observable markov decision process (pomdp) (g, s, o, f, p, z, r). g is a set of goals, which we refer to as "tasks", that the user could ask. (e.g. "reply to my last email from dev with the body 'let us catch up soon to discuss the project.' " 2 ); state @xmath0, emails, calendar events, etc.; o is the information the agent currently knows (e.g. dev's email is dev@business.com); f is the set of function names (e.g. reply_email) the agent can choose to execute; p (f ) is the parameter schema given whose domain contains the valid inputs of @xmath1.g. email id and message). z represents the transition dynamics. executing function f and input parameters @xmath2, o ′ = z(s, f, p) (e.g. s ′ is dev having a new message in his inbox and o ′ is the agent receiving a message that the email was sent). finally, r(s ′ |g) : s × @xmath3, 1}, where @xmath4. note that r is not dependent on o ′ as the observation may not correctly indicate task completion (e.g. "email sent successfully" is returned to the agent, but the email is sent to the wrong person.) in this work, the policy π of that api-based agent that completes tasks is a fixed llm 3 . for an llm-based agent to understand the functions in f, it needs a textual description, or textualization, of p , and z (e.g., api documentation). let t * p , t * z be the set of all possible textualizations of p and z, respectively, that correctly convey the functionality of f. with t @xmath5, t @xmath6, an agent selects an api with input parameters with a policy (f, p) ∼ π(•|o, f, t p , t z , g). in practice, the llm agent generates a json object detailing the chosen f and p that will be passed to another program to execute. note that π is treated as a probability due to the inherent randomness of llms. however, given the agent has access to some t p and t z , it can make appropriate decisions. limitation in prior work: access to t p and t z . as mentioned previously, api documentation is often inaccessible, out-of-date, or inconsistent with the functionality of the apis. thus, the agent may not have access to some t p and t z , and we aim to learn and model the information as t θ p and t θ z for p and z, respectively. ideally t θ @xmath7. we wish to maximize the total successful task completions with (t θ p , t θ z ). letting h be the number of steps an agent performed to try to complete g, we can formally write equation 1 provides a mathematical grounding of how well t θ p and t θ z textualize the unknown p and z to how well the llm-based agent performs the set of tasks g. the next section details how we tackle this problem by learning θ via demonstrations of api function calls. ## learning of api functionality from in-context demonstrations although api documentation may inaccessible, often there are demonstrations available based on codebases of public repositories and logs. with this in mind, we investigate how we can utilize 3 we will use the terms "agent" and "policy" interchangeably; so π refers to both. ## experimental setup in this section, we provide the experimental details for studying our methods of learning of api functionality from in-context demonstrations. across our experiments, we run 3 trials, each with a different random seed @xcite . ## results: learning parameter information is crucial from our experiments, a recurring problem at every stage of the pipeline (document generation, task completion, and evaluation) is the parameter fill13 in the appendix). other sources of error were more specific, such as when generated documents would specify to include timezone information for time_min and time_max parameters in workbench functions. agents that followed this got a runtime error. the generated documents from more demonstrations would more frequently make this mistake, and also say that the time_min and time_max parameters were required when they were optional. documents generated from fewer demonstrations would, in some trials, correctly say that those parameters were optional, allowing the agent to not fill in those parameters with incorrect formatting. we hypothesize that the more repeated use of these parameters in the larger set of demonstrations causes the document generator to believe that those are required, causing the degradation in generated guidelines can help parameter understanding. attaching generated guidelines (ag) has a 12% improvement over (gd) in workbench. in a representative case, "kerry moore is no longer a customer. can you delete them from the crm?", the gd agent tries to directly delete the customer by using their name as the customer_id parameter, rather than searching for their id and then deleting. the guidelines for customer.delete_customer state to avoid directly deleting customers and ensure that the customer id is a valid value, not someone's name. this result shows that selfexploration with an evaluation protocol can enable self-improvement. error handling of parameter formatting affects performance. however, we see that for τ -bench, ag, and all other methods except for od, achieved a low score with τ -bench. a major contributing factor is the order_id parameter. the or-der_id parameter is in 7 out of 14 functions in τ -bench, and requires a '#' at the beginning of the value; otherwise, an error is returned. when the order_id is formatted incorrectly, the error message "order not found", shown in figure 6 , is misleading to the agent and the self-improvement system because there is no indication that the missing "#" is the problem of the error. some of the generated evaluations marked some of the function calls that used the '#' as wrong. this error in evaluation then affects the summarized guidelines. in 2 trials, the guidelines would say to remove the '#' formatting from ids. in τ -bench, none of the generated documentation methods (gd, ud, and rd) mention the formatting requirement either. therefore, the only way to know that the "#" is required was by using the od. we reran the ag (with dxd method before self-exploration) experiment. however, this time we hardcoded the following phrase in all the guidelines for all functions that use order_id: "there needs to be a '#' at the start of the order id. this is mandatory. for exam-ple: "#w8732376" is valid, but "w8732376" is not!!!!" this modification raises the average success rate from 10% to 49%, and this result shows again that having specific, correct parameter information can significantly help the sr, and that robust and descriptive error handling of the api functions is imperative for reliable evaluation and summarization. dxd and de have more advantages than the summarization methods. dxd and de have consistently had a higher success rate over other i and i ′ methods. however, this can be attributed to data leakage. the set of tasks g for each environment has many repeated formats, such as sending emails, canceling orders, or looking up information on cases. thus, the train-test distributions are similar. it would be interesting to try dxd and de in environments that provide distinctive tasks that use the same apis. note on increasing n . for dxd with τ -bench and crmarena, using the conversation history between causes the llm of the agent to exceed its maximum context length. we truncated the conversation history to at most the past three messages. this truncation allowed o3-mini to handle 5 and 15 demonstrations per function for τ -bench, and 5 per function for crmarena. all other models could only handle 5 demonstrations per function. for gdec, we noticed that for workbench and crmarena, the agent will repeatedly call the same function, suggesting that the repeated n example calls teach the llm agent to repeat. ## related works tool-based agents expand the capabilities of traditional llms by connecting to external sources such as api functions, search engines, data centers, etc., to complete multi-step queries. multiple benchmarks have been proposed with various sets of api functions with documentation @xcite @xcite @xcite @xcite . while generating documentation has been studied before the advent of llms @xcite , there is little work on generating documentation for downstream agent task planning. for documentationfree agents, toolalpaca by @xcite generated documentation via an llm given a brief description of the function from openapi. api-docbooster @xcite used gpt-4 to produce and update documentation based on search results from stackoverflow. concurrent to this work, @xcite also used experiences of the agent to update the documentation. however, they still relied on initial documentation. toolformer by @xcite finetuned their agent on api calls. our work is the first to study how we can generate and update an agent's understanding of functionality with expert demonstrations and experience with a frozen llm. we are also the first to use demonstrations directly into the llm-based agent for functionality understanding. our formalization is similar to reinforcement learning with parameterized action spaces @xcite where the agent must choose an action from a discrete set (in our case, choose an @xmath8, and then must choose parameters specific to the selected action, i.e. @xmath9. however, to our knowledge, previous literature assumes a single, continuous parameter per action. our work relaxes this assumption as an api function can allow for multiple parameters, continuous or discrete. ## conclusion we provide and formalize the problem of earning of api functionality from in-context demonstrations with no prior documentation. we highlight the persisting and challenging problem of api-agent planning with limited information on the set of functions. we investigate tackling this problem by learning functionality, specifically function description, return, and input parameters, from in-context demonstrations of tool calls. we analyze the number of demonstrations, various processing methods, and the impact of self-exploration and llm-based evaluation. importantly, our extensive experiments highlight the difficulties of sota llms on this problem, specifically due to failures in describing the parameter schema, suggesting further research to improve results.