API - BLEND : A Comprehensive Corpora for Training and Benchmarking API LLM s (2024)

Abstract: There is a growing need for Large Language Models (LLMs) to effectively use tools and external Application Programming Interfaces (APIs) to plan and complete tasks. As such, there is tremendous interest in methods that can acquire sufficient quantities of train and test data that involve calls to tools / APIs. Two lines of research have emerged as the predominant strategies for addressing this challenge. The first has focused on synthetic data generation techniques, while the second has involved curating task-adjacent datasets which can be transformed into API / Tool-based tasks. In this paper, we focus on the task of identifying, curating, and transforming existing datasets and, in turn, introduce API-BLEND, a large corpora for training and systematic testing of tool-augmented LLMs. The datasets mimic real-world scenarios involving API-tasks such as API / tool detection, slot filling, and sequencing of the detected APIs. We demonstrate the utility of the API-BLEND dataset for both training and benchmarking purposes.

## introduction large language models (llms) have shown remarkable abilities across a variety of natural language understanding (nlu) tasks @xcite , e.g., text generation @xcite , summarization @xcite , and mathematical reasoning @xcite . there has been strong recent interest in enabling llms to call apis or external tools (such as calculators, calendars, or web searches @xcite @xcite ) to accomplish high level tasks like booking a hotel, reserving a table, and automating a job requisition tasks. these higher-level tasks are generally conversational and complex. however, in order to perform such complex tasks, llms should be able to perform simpler tasks with apis such as (a) apis detection: based on a user query, correctly choose which api to call, (b) slot filling foot_0 : given the api, extract either the slots/parameters from the user utterances or request from the user more information to fill the required parameters of the detected api, and (c) sequencing: given an utterance specifying a task, write the sequence of apis that needs to be called to accomplish the task. data for the above-mentioned api tasks, both for training and benchmarking llms has been scarce. addressing this issue, in the recent past, most approaches have relied on synthetically generated data for training api-augmented llms. for instance, toolllm @xcite produces multi-sequence rest-api data sourced from gpt-4 @xcite , while datasets like gorilla @xcite utilize synthetic, singlesequence api data, specifically deep learning libraries' apis, generated from language models. although generating synthetic data from llms offers a cost-effective means of obtaining substantial training data, they suffer from several drawbacks. first, data generation methods are plagued by critical issues such as bias inherent in the sampling model, and a lack of diversity in the training dataset. previous work has shown synthetically generated data suffers from lack of diversity @xcite , and diverse data can improve out-ofdomain (ood) generalization @xcite . consequently, models trained on such data can overfit on in-distribution apis and struggle to generalize to ood apis that were not encountered during training, restricting the broad applicability of llms for tool usage. in addition, datasets have primarily included api detection (single and multiple) and slot filling in different settings, whereas sequencing, a prominent task to perform higherlevel human tasks using apis has rarely been the focus in existing works. lastly, datasets in domains such as digital assistants and semantic parsing that are related to api-tasks and are human-annotated have gone unnoticed in literature due to the emergence of synthetic data generation techniques; in light of the above issues, we have developed an api-focused training dataset that leverages a hybrid approach for data generation. this is built upon five human-annotated datasets with llmassisted generation comprising of over 150k, 17k, and 11k train, development, and test instances. the transformation primarily focuses on sequencing, including api detection and slot-filling due to the sparsity and importance of sequencing data for training models. furthermore, these datasets are collected from different domains such as semantic parsing, dialog, and digital assistants resulting in a higher diversity of api data. we show that models trained on this diverse dataset yield significantly better ood generalization performance compared to other state-of-the-art methods, with an example shown in figure 1 . as an evaluation/benchmarking dataset, we include five different existing benchmarks for ood evaluation. in conclusion, we release api-blend, a comprehensive api training, and benchmarking dataset, comprising of 10 datasets (5 for training, and 5 for ood testing), see figure 2 . ## api-blend dataset curation we focus on the setting where the input is a single natural language utterance and the output is a sequence of api calls with their parameter names and values. api-blend consists of datasets created via the following three approaches: (1) language model assisted approach where prompts are used based on existing api outputs, (2) a grammar rule-based approach to convert existing semantic parsing and personal assistant notations into api data, and (3) off-the-shelf datasets. ## data quality assessment to check the quality of api-blend datasets, we perform human annotation over 50 randomly sampled data from 5 in-distribution datasets (at least 2 annotators per dataset). the following steps are taken to measure the data quality: first, the annotators score each gold sample (1 if correct, 0 otherwise) based on the following 3 metrics: (1) "intent": whether the intents/apis in the output are correct to answer the query; (2) "sequence(intent)": whether the intents are in the proper sequence; and (3) "slot": whether the slot names and slot-values are correct. then, we calculate the average scores across all the annotators and present them under the "data quality" section in as the api-blend data suite has been generated from high-quality datasets from different domains, we are seeing very high scores in our data quality assessment, particularly for the easier "intent" and "sequence(intent)" tasks. ## conclusion this paper presents api-blend, a large corpora for training and evaluation of tool-augmented llms, curated from real-world datasets of different domains such as dialog, semantic parsing, digital assistants, etc. api-blend consists of 10 datasets (5 in-distribution and 5 out-of-distribution) comprising over 190k instances (including train, development, and test). we have demonstrated that the models fine-tuned with api-blend generalize better than the other tool-augmented llms on ood experiments. our findings not only substantiate the importance of api-blend in training and benchmarking tool-augmented llms but also highlight the necessity of generalizability to improve the api usage capability of llms.