Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks (2024)

Abstract: An emergent research trend explores the use of Large Language Models (LLMs) as the backbone of agentic systems (e.g., SWE-Bench, Agent-Bench). To fulfill LLMs’ potential as autonomous agents, they must be able to identify, call, and interact with a variety of external tools and application program interfaces (APIs). This capability of LLMs, commonly termed function calling, leads to a myriad of advantages such as access to current and domain-specific information in databases and the outsourcing of tasks that can be reliably performed by tools. In this work, we introduce Granite-20B-FunctionCalling, a model trained using a multi-task training approach on seven fundamental tasks encompassed in function calling. Our comprehensive evaluation on multiple out-of-domain datasets, which compares Granite-20B-FunctionCalling to more than 15 other best proprietary and open models, shows that Granite-20B-FunctionCalling has better generalizability on multiple tasks across seven different evaluation benchmarks. Moreover, Granite-20B-FunctionCalling shows the best performance among all open models and ranks among the top on the Berkeley Function Calling Leaderboard (BFCL).

## introduction function calling provides a means for language models to leverage external tools and resources. these tools can make available to an llm specific, up-to-date information that would otherwise be inaccessible (e.g., stored in a dynamic knowledge base) and thus reduce its proclivity for hallucinating responses @xcite . this is particu-⋆ these authors contributed equally to this work † corresponding authors: {ibrahim.abdelaziz1, kinjal.basu, mayank.agarwal}@ibm.com, kapanipa@us.ibm.com 1 the model is available at: https://huggingface.co/ ibm-granite/granite-20b-functioncalling larly crucial in enterprise use cases where a significant portion of relevant data is stored in a structured format accessible only via storage engines. in addition to knowledge access, function calling can allow an llm to outsource tasks that are out of scope for a generalized language model. most commonly, these tasks involve compute-heavy operations, e.g., program execution @xcite , numerical calculation, or retrieval @xcite , and are otherwise a frequent source of llm hallucinations @xcite . the importance of function calling has spurred the development of several recent data generation efforts for fine-tuning @xcite @xcite @xcite and evaluation of models @xcite . however, the fine-tuned models from datasets like toolllm @xcite , toolalpaca @xcite , and gorilla @xcite ) fall short in one (or more) of three key dimensions: (a) generalizability: while the datasets are generated using diverse sets of apis (e.g., toolllama uses rapidapis foot_0 , toolalpaca uses public apis foot_1 , and gorilla uses tensorflow hub, pytorch hub, and hugging face hub), basu et al. ( 2024 ) has shown that models trained on these datasets have difficulty generalizing to out-of-domain datasets. (b) ability to handle granular tasks: function calling, as an umbrella term, encompasses multiple granular sub-tasks such as function-name detection, slot filling foot_2 or parameter-value pair detection, and detecting the ordered sequence of functions needed to be called. existing models trained to perform function calling lack the ability to handle these granular tasks independently, and hence, perform poorly on such sub-tasks. (c) openness: the best performing models are proprietary and the ones that have open licenses (e.g., @xcite ) are trained using data generated from openai models. our work addresses these limitations, and centers on introducing function-calling abilities to models with an inherent focus on granular tasks. figure 1 shows an overview of how granite-20b-functioncalling was trained. our work draws largely on data obtained from api-blend @xcite , which comprises the following tasks: function name detection, slot filling, parallel functions, multiple functions, sequencing, 5 and calling apis foot_4 using multiple programming languages. we build upon granite code models @xcite by instruction tuning them for function calling using the datasets for granular tasks with a multi-task learning approach. we perform a comprehensive evaluation of the open and proprietary models using bfcl, four function calling academic benchmarks, and the response generation benchmark @xcite to evaluate the generalizability of function-calling models. granite-20b-functioncalling is on par with the best open model on bfcl and ranks fourth overall. furthermore, granite-20b-functioncalling exhibits superior generalizability over other models on the out-of-domain datasets. figure 2 shows how granite-20b-functioncalling compares to the top two open models (according to bfcl) on various tasks where despite only having 20b parameters, it performs as well or better than meta-llama-3-70b-instruct which has 70b parameters. ## multi-task training data in this section, we outline our comprehensive approach to curate multi-task function calling data to fine-tune granite-20b-code-instruct @xcite model, thereby creating our robust granite-20b-functioncalling model specifically designed for function-calling. our training data mixture draws largely on api-blend @xcite , which compiles five datasets (seqsgd, seqsnips, seqtopv2, se-qatis, and seqmultiwoz) totalling about 160k training examples. in addition, we also use the glaive-v2 foot_5 dataset. as a data pre-processing step, we unify the format of all these datasets. a key contribution to the process of building granite-20b-functioncalling is multi-task training, where we reuse the same data in different formats with distinct instructions for different function-calling related tasks. we identified six underlying sub-tasks for function calling and divided them into two broad categories based on their respective difficulty levels: (a) low-level function calling tasks which are simpler tasks for an llm and relate to either function names or only parameter-value pairs; and (b) high-level function calling tasks which are complex tasks for an llm and typically handle multiple functions; to excel in high-level function calling tasks, it is crucial for any llm to master the low-level foundational sub-tasks. we have included "response generation" as the seventh task in our training data since producing natural language responses is one of the fundamental goals of an llm. ## experimental setup and evaluation in the section below, we detail our extensive evaluation on various evaluation datasets and public leaderboard. we provide a comprehensive comparison of our granite-20b-functioncalling to other open and proprietary function calling models. ## conclusion in this paper, we introduced granite-20b-functioncalling, a capable function calling open model with apache 2.0 license. it is trained using a suite of datasets transformed from different domains and with a multi-task learning approach. we performed an extensive evaluation of our model in comparison to other state-of-the-art function calling models. on multiple out-of-domain datasets, including bfcl, our model outperform the other open models. even compared to multiple proprietary models with much larger sizes, our model showed on-par and in some cases better performance on multiple datasets and tasks.