FANTA stic SE quences and Where to Find Them: Faithful and Efficient API Call Generation through State-tracked Constrained Decoding and Reranking (2024)

Abstract: API call generation is the cornerstone of large language models’ tool-using ability that provides access to the larger world. However, existing supervised and in-context learning approaches suffer from high training costs, poor data efficiency, and generated API calls that can be unfaithful to the API documentation and the user’s request. To address these limitations, we propose an output-side optimization approach called FANTASE. Two of the unique contributions of FANTASE are its State-Tracked Constrained Decoding (SCD) and Reranking components. SCD dynamically incorporates appropriate API constraints in the form of Token Search Trie for efficient and guaranteed generation faithfulness with respect to the API documentation. The Reranking component efficiently brings in the supervised signal by leveraging a lightweight model as the discriminator to rerank the beam-searched candidate generations of the large language model. We demonstrate the superior performance of FANTASE in API call generation accuracy, inference efficiency, and context efficiency with DSTC8 and API Bank datasets.

## introduction in recent year, there has been a surge of interest in enabling the automated tool-using capability of intelligent systems @xcite . specifically, as a bridge to the larger world, application programming interface (api) calls allow virtual assistants to control smart-home devices, retrieve information, make reservations, and more on the user's behalf. figure 1 shows how an api call may improve the user-assistant conversation and satisfy the user's needs. generating such an api call requires advanced capabilities in understanding the requirements of an api (including † the major portion of the research was done during an internship at amazon. its endpoints, parameters, and expected data formats) and reasoning over the conversation context to translate the user's needs into the appropriate api format. with recent breakthroughs in generative large language models (llms) such as gpt-x @xcite and llama @xcite , researchers have started to investigate their competence in complex reasoning tasks such as utilizing appropriate api tools @xcite @xcite . their attempts focus on methods that can generally be grouped into those based on supervised fine-tuning for task-specific usage and those based on augmenting input-side context information (such as api shortlisting and exemplar selection) and optimizing prompts for in-context learning @xcite . despite the strong supervision or extensive context, these methods still cannot ensure the generation's faithfulness with respect to the api documentation and suffer data and compute inefficiency. in contrast to previous works, we focus on how decoding strategies improve the generation's faithfulness, which is complementary to supervised fine-tuning and in-context learning methods. as a result, we present fantase (fan-tastic sequences and where to find them), a framework that employs state-tracked constrained decoding (scd) and reranking components, for faithful and efficient api call generation. the scd tracks the states of the generation and retrieves appropriate api documentation constraints in the form of constrained token search trie (ctst) used at each decoding step. scd is guaranteed to generate api calls that are faithful with respect to the api documentation ( § 4.1), and provides inference efficiency ( § 6.2) with ctst that eliminates unnecessary forward inference passes. compared to supervised fine-tuning methods, scd brings considerable improvements ( § 6.1) without the data labeling and model training related hefty costs of labor, time, and computing that become increasingly expensive as the size of the llm grows @xcite . scd also reduces the in-context learning's reliance on the repeated supply of extensive contextual information for the inference of each instance ( § 6.3) by effectively incorporating api documentation constraints and guaranteeing the associated faithfulness at the decoding stage. the reranking component of fantase leverages models that are significantly smaller than llms for efficient incorporation of supervised signals ( § 4.2). as the correct api generation may not always have the highest sequence probability among beam-searched candidate sequences ( § 3), we train lightweight models to discriminate and rerank llms' candidate generations and demonstrate their effectiveness in digging out those correct sequences ( § 6.1). compared to the supervised fine-tuning of llms, the reranking component features extremely low training costs as it employs lightweight models. compared to input-side optimized in-context learning methods, the reranking component can address the severe performance issue associated with the absence of valuable supervised signals. notably, fantase is a approach that suits the evolving and vast nature of real-world apis. with the update of api documentation or the application to the new domain, llms fine-tuned with old data would require re-tuning with new data @xcite . for fantase, scd can adapt by constraining the decoding with a new set of constraints elicited from the new api documentation, while re-tuning the lightweight reranking models has lower time and compute cost. in summary, we make the following novel contributions: constrained decoding offers controllable text generation by enforcing certain constraints at the decoding stage. early research (hokamp and liu, 2017; post and vilar, 2018) concentrated on lexical constraints that enforce the inclusion of specific words or phrases in the outputs, which often neglects broader syntactic or semantic relationships. later on, lu et al. 2021 introduced neurologic decoding that handles more complex lexical constraints expressed by predicate logic. the subsequent extension, neurologic a*esque decoding (lu et al., 2022), incorporated a lookahead heuristic to estimate future lexical constraint satisfaction. more recently, chen et al. 2022 and bastan et al. 2023 proposed parsing-based constrained decoding algorithms that tackle the challenge of ensuring correct syntactic relationships between word pairs. specific to structured text generation, scholak et al. 2021 targeted text-to-sql generation and introduced picard that checks the validity at each decoding step for sql lexical and grammar correctness with incremental parsing. the latest advancement was made by geng et al. 2023 who demon- ## preliminary analysis to better understand the capabilities and limitations of existing llms on the task of api call generation, we conduct a preliminary inference analysis on one hundred dstc8 @xcite foot_0 samples with an alpaca @xcite model that had been tuned with gpt-generated self-instruct @xcite ) data for better instruction following and in-context learning capabilities. we prompt the model with the dstc8 data that contains task instruction, documentation of related apis, two related exemplars, and conversation history. we use beam search with beam size 10 as the decoding algorithm, and we consider the top-10 high probability sequences as the candidate generations. our quantitative analysis shows that for 73% of the cases, the correct api calls are generated within those high probability sequences. however, within these cases, almost half of the correct sequences were not ranked as the highest, which yields a top-1 api call generation accuracy of 41%. we conduct a further qualitative analysis to categorize the error types and possible mitigation for these cases. for the highest-ranked error cases, we find 33% argument value error, 24% missing required arguments, 19% missing optional arguments, 14% hallucination, and 10% argument name error. furthermore, 42% of the error cases can be mitigated by enforcing the constraints described in the api documentation, 29% of the cases require the better understanding of the conversation, and the remaining 29% of the cases need a combination of the aforementioned two improvements. ## the fantase framework in section 3, we demonstrate that there are "fan-tastic sequences" in the beam-searched candidate generations, and the question is where to find them. to dig out those "fantastic sequences" with the data efficiency and compute efficiency in mind, we propose the fantase framework that consists of two major components -state-tracked constrained decoding ( §4.1) and reranking ( §4.2), which aims at enforcing api constraints with guaranteed faithfulness to the api documentation and incorporating supervised signals at low compute costs respectively. ## conclusions the fantase framework, with its state-tracked constrained decoding (scd) and reranking components, effectively tackles the challenges of generating api calls from complex contexts. by integrating api constraints through a token search trie and employing a lightweight model for reranking, fantase not only ensures accurate api call generation but also improves inference and context efficiencies. its superior performance on the dstc8 and api bank datasets confirms fantase's significant advancement in enhancing large language models' tool-using ability.