Sequential API Function Calling Using G raph QL Schema (2024)

Abstract: Function calling using Large Language Models (LLMs) is an active research area that aims to empower LLMs with the ability to execute APIs to perform real-world tasks. However, sequential function calling using LLMs with interdependence between functions is still under-explored. To this end, we introduce GraphQLRestBench, a dataset consisting of natural language utterances paired with function call sequences representing real-world REST API calls with variable mapping between functions. In order to represent the response structure of the functions in the LLM prompt, we use the GraphQL schema of the REST APIs. We also introduce a custom evaluation framework for our dataset consisting of four specially designed metrics. We evaluate various open-source LLMs on our dataset using few-shot Chain-of-Thought and ReAct prompting to establish a reasonable baseline.

## introduction tool use in large language models (llms) is an active area of research that aims to overcome the limits of pretraining llms (which usually results in a "knowledge cutoff date") by enabling the llms to fetch data that they were not trained on using tools such as web apis and databases. in this context the idea of using llms for function calling has gained traction since using tools in the form of functions requires llms to accurately pass correct parameter values to the functions. any web api can be encapsulated as a function which requires inputs in a predefined format and outputs a structured response object. the idea of empowering llms to use tools to harness external knowledge and perform complex computational tasks was introduced by toolformer @xcite . there have been several attempts to train llms to use tools such as apis @xcite @xcite @xcite . llms still do not perform well on api calling due to their inability to generate accurate input arguments and their tendency to hallucinate the wrong usage of an api call. it is essential for api-augmented llms to have robust planning and decision-making capabilities. planning based approaches like react @xcite encounter challenges in effectively adapting api feedback and generating viable plans. restgpt @xcite introduced a coarse-to-fine online planning mechanism for task decomposition and api selection, and api execution. while methods like react and restgpt have demonstrated promising abilities for online planning and execution, they may generate incorrect apis during the exploration phase. in contrast, @xcite focuses on the ability of the llm to call a given api correctly. we wish to ex-tend this approach to the sequential api execution scenario of restgpt. while the gorilla openfunctions framework (see the berkeley function calling leaderboard @xcite ) supports single and parallel function calls, it does not as yet support the use case of chained or sequential function calls where there exist mappings between the input and output parameters of functions. the fundamental difficulty in calling sequential apis in a single shot is the lack of knowledge about the response structure of apis. while the openapi specification of the api might provide some clue as to the response structure, it is often incomplete or inadequate for the purpose of defining the variable mapping in pythonic form. graphql (inc., 2015) is a query language for apis that allows the user to easily find the useful fields and types in the api response object by inspecting the so-called graphql "schema" of the api using a feature called "introspection". as a solution to the above problem, we propose using the graphql schema of the apis as a reliable source of information regarding their response structure. tools like stepzen (ibm, 2024), apollo (apollo graph inc, 2024), and hasura @xcite are available for automatically generating the graphql schema for querying restful apis and databases. in this paper, we introduce a new dataset, graphqlrestbench foot_0 which is built using the restbench dataset introduced by restgpt. notably, restbench only provides api sequences and not input-output parameter mappings between apis. it can therefore be used only for measuring whether the generated sequence of function names is exactly the same as the ground truth sequence. in graphqlrestbench, we additionally add the graphql schema generated by stepzen for the apis and also python code to call the apis in a sequence using input-output parameter mapping given the response structure of the apis obtained from the graphql schema. the task is to generate the correct python code consisting of a sequence of function calls with accurate parameter mapping between functions (see figure 1 ). sometimes the model may generate a different sequence of function names which is still meaningful for the task because the input-output dependence between the function calls is preserved. furthermore, we have only considered required parameters for the apis in the ground truth function calls. but the model may generate additional optional function arguments which are not actually required by the apis to return the correct response, but are still present in the input signature. keeping this in mind, we introduce a custom evaluation framework for our dataset consisting of four task-specific metrics. we also evaluate various open source llms on this task using chain-of-thought @xcite and react @xcite style prompting as a reasonable baseline. the concept of using graphql to represent the output signatures of the functions was not in rest-bench or in any other function calling dataset such as the berkeley function calling leaderboard dataset. we would like to claim this as the main contribution of our work. the provided graphql schemas are useful for interacting with the apis through a graphql interface rather than through a standard http request. agent-based frameworks have also been explored in this area. react @xcite studied the integration of reasoning and acting (by means of function calls) in llm agents. inspired by re-act, restgpt @xcite proposes a dualagent planner-executor approach to connect llms with real-world restful apis. @xcite introduced exploration-based trajectory optimization for open-source llm agents by fine-tuning on the agent trajectories. anytool @xcite introduced self-reflective, hierarchical agents for api calling using the function calling ability of gpt-4 @xcite . hugginggpt @xcite is an llm-powered agent that connects various ai models in machine learning communities such as hugging face to solve ai tasks. restful is the popular web service development standard @xcite , which supports http protocols and uris to serve resources. ope-napi specification (initiative, 2021) describes the operations, parameters, and response schemas in restful apis. function calling datasets apibench from @xcite consists of huggingface, torchhub, and tensorhub apis. restbench from restgpt @xcite consists of apis from tmdb movie database and spotify music player. toolbench from toolllm @xcite consists of 16,464 real-world restful apis spanning 49 categories from rapidapi hub. anytoolbench from anytool @xcite is similar to tool-bench but with a different evaluation protocol. graphql @xcite discussed generating graphql wrappers for rest apis using the openapi specifications. @xcite proposed automatic graphql schema generation for data-intensive web apis using a semantic metamodel. works such as (brito and valente, 2020) compare graphql and rest frameworks. ## methodology in this section we explain the methodology we used to create the graphqlrestbench dataset. graphql schema generation first we generate graphql schema for all the api endpoints in rest-bench, except for those whose output schema is never required. we use the import curl command from the stepzen cli to generate the graphql schema for the endpoints using appropriate dummy values for the parameters if required. the schema files thus generated are collated to form the combined schema for a given sample (sequence of api calls) in restbench. function signature generation we programmatically generated function signatures in the openai compatible format used by gorilla openfunctions @xcite and the berkeley function calling leaderboard @xcite by parsing the openapi specifications for spotify and tmdb available in restbench.