Adapting LLM s for Structured Natural Language API Integration (2024)

Abstract: API integration is crucial for enterprise systems, as it enables seamless interaction between applications within workflows. However, the diversity and complexity of the API landscape present significant challenges in combining API calls based on user intent.Existing methods rely on named entity recognition (NER) and knowledge graphs, but struggle to generate more complex control flow structures, such as conditionals and loops.We propose a novel framework that leverages the success of large language models (LLMs) in code generation to integrate APIs based on natural language input. Our approach involves fine-tuning an LLM using automatically generated API flows derived from OpenAPI specifications.We further evaluate the effectiveness of enforcing the syntax and schema adherence through constrained decoding.To enable systematic comparison, we introduce targeted test suites to assess the generalization capabilities of these approaches and their ability to retain structured knowledge.Our findings show that LLMs fine-tuned on OpenAPI specifications can (a) learn structural API constraints implicitly during training, and (b) achieve significant improvements in both in-distribution and out-of-distribution performance over NER and retrieval-augmented generation (RAG)-based approaches.

## introduction the ability to integrate apis of different software services is crucial for automating processes across applications. industrial tools like ibm app connect foot_1 or zapier foot_2 provide visual interfaces for manual flow composition but they require users to possess api knowledge or tediously search through service catalogs. this motivates automatic flow generation from natural language descriptions. gofa @xcite demonstrates the feasibility of such solutions by implementing utterance-to-api generation with an ner-based approach. gofa, however, struggles with variations in user utterances and limited support for complex flow control structures like conditionals and iterations, as they require more complex reasoning over the natural language query. recent successes of large language models (llms) on related code generation tasks like text-to-sql @xcite @xcite encourage exploring their capabilities for this task. this requires the llm to learn (a) mapping utterances to relevant apis, (b) valid methods within those apis, and (c) the syntactical constraints for composing api flows. to this end, we propose a generic llm-tuning approach where structured information is (a) implicitly learned through automatically generated samples and (b) optionally enforced at inference time with constrained decoding. our contributions are summarized as follows: ## automatic data generation our goal is to automatically generate training data for llm text-to-api flow fine-tuning, thereby aligning the model with api domain knowledge. to this end, we leverage the tree structure and node attributes of the api forest depicted in figure 1 . prompting with this large structured information is still difficult. firstly, despite efforts to increase prompt length for modern llms (e.g., longllama; @xcite , maximum token limitations restrict the amount of structured information that can be passed during inference. further, llms have been shown to perform worse for longer prompts due to the amount of irrelevant context @xcite . to this end, a common mitigation strategy is retrieving-augmented generation (rag) @xcite . however, as shown in section 5, rag has limited impact on domain-specific tasks where semantic search over unseen concepts performs poorly. therefore, we train models to learn structural knowledge implicitly through generated training samples. generating samples manually is expensive and requires specific api flow expertise. we therefore propose a synthetic data generation approach to create a rich training set capturing diverse utterance-to-flow pairs. namely, we first extract openapi descriptions from the "description" and "responses" attributes of object methods (examples 1 & 2 in figure 2 ). as these descriptions vary in quality, we additionally generate operation-specific templates filled with api details from the specification (examples 3 & 4 in figure 2 ). this aims to generate a diverse set of api call descriptions covering potential user utterances. matching utterance intent to the appropriate api call requires considering the limited set of request methods. the same method can have different meanings depending on the endpoint it calls. for instance, the create method on a gmail message object sends an email, while on a salesforce account, it creates the account. matching intent might require more than simple semantic parsing and should incorporate api descriptions. { info: ... base_path: /gmail/ paths:{ /mail:{ get: { description: ... summary: ... operationid: create_mail } ... } we use the following categories for generating training samples, representing the building blocks of api flows: we provide examples for each category in to mitigate spurious correlations and improve outof-distribution generalization, we augment data with paraphrasing-a common technique used to increase data variability @xcite @xcite . we paraphrase the synthetically generated samples using few-shot prompting. finally, we filter out samples where application or object names are lost during paraphrasing using partial string matching. ## enforcing valid api flows event-driven api integrations can be represented by a constrained subset of python code with api calls expressed as app.object.operation triplets. the precise grammar, which also directly encodes the set of available apis, is specified in ebnf-like notation. for completeness, the grammar is shown in figure 3 . note, that the schema is encoded in the set of terminals named actions and triggers, resulting in a rather large grammar. during incremental parsing, such terminals are split up to match the current generation with valid matching suffixes at each decoding step. to enforce adherence to this structured knowledge and syntax, we employ constrained semantic decoding @xcite at inference time, as even tuned models can deviate from valid schemas, especially in ambiguous scenarios. we adopt a faster implementation of constrained semantic decoding for beam search. instead of building prefix trees to find all valid continuations at each step, we directly check whether the continuation token is valid on the most likely tokens until we find k valid continuations. since beam search samples generate at most k continuations per beam, we only need to keep the k most likely valid tokens per beam, i.e., its beam width. this significantly reduces the number of expensive isvalidprefix calls. the procedure is shown in algorithm 1. algorithm 1 fast constrained decoding require: prefix p, grammar g, tuned model p θ , beam width k ensure: masked next token scores 1: l ← sortprob(getnexttokenprob(p, p θ )) 2: ct, validt okens ← 0, [] 3: for tok in l do 4: while @xmath0: if isvalidprefix(p • tok, g) then 6: append(validt okens, tok) 7: ct ← ct + 1 8: end if 9: ## evaluation we evaluate our approach by training several llms and comparing them to an ner baseline. all models are trained on the same apis with 85 applications, 4'557 application-specific objects, and 21'712 unique api calls. as each application has an arbitrarily large number of objects/endpoints and each endpoint may support a different subset of the actions and triggers, the resulting trees are much differently sized. this is shown in figure 4 . ## related work database and knowledge graph querying are wellknown nlp problems, often addressed through techniques such as ner, relation extraction, and query generation. these methods typically involve producing graph queries from natural language utterances and executing them against graph databases @xcite @xcite @xcite . modern database interfaces also employ llms for converting user queries into sql @xcite . web api search using deep learning models has been explored by @xcite . this work proposes synthetic dataset generation and leverages deep learning for api integration. however, the approach relies on substring detection routines, which can be less flexible than the llm-based adaptation presented here. further, gorilla @xcite is an llm trained to access apis for interacting with ml models on platforms like torch hub, ten-sorflow hub, and huggingface. it excels at interacting with individual models but is not specifically tuned creating flows between apis. ## limitations a limitation of the proposed approach is that this work focuses solely on generating flows of api call triplets. as such, the trained models do not generate arguments for the api calls (cf. appendix b). however, we note that including arguments in a written utterance is practically rather tedious and a semi-supervised approach may be better suited to address this need. ## conclusion this work presents a novel approach to natural language-driven large-scale api integration using llms. we demonstrate that models trained with our approach exhibit strong generalization capabilities, both in-distribution and out-of-distribution. this is evident in their ability to: (a) handle ambiguity by leveraging structural knowledge to make informed decisions when user intent is unclear; (b) learn domain knowledge by adapting to domainspecific phrasing and terminology encountered during training; and (c) generate unseen flow structures by utilizing the capability of general-purpose llms, particularly llama, to create novel api flow compositions that adhere to implicit syntactic constraints. these findings highlight the potential of llms to streamline api integration tasks.