Octopus: On-device language model for function calling of software API s (2025)

Abstract: Large Language Models (LLMs) are pivotal for advanced text processing and generation. This study presents a framework to train a series of on-device LLMs optimized for invoking software APIs. Using a curated dataset of 30,000 API function calls from software documentation, we fine-tune LLMs with 2B, 3B, and 7B parameters to enhance their proficiency in API interactions. Our approach improves the understanding of API structures and syntax, leading to significantly better accuracy in API function calls. We also propose a conditional masking technique to enforce correct output formats, reducing errors while maintaining inference speed, specifically tailored for API tasks. The fine-tuned model, Octopus, outperforms GPT-4 in API calling tasks, showcasing advancements in automated software development and API integration. The model checkpoints are publicly available.

## introduction the advent of large language models (llms) has revolutionized artificial intelligence, enabling transformative applications in natural language processing and specialized domains such as mathematics @xcite , healthcare @xcite , and legal analysis @xcite . despite their advancements, llms face challenges in adapting to real-time updates and performing domain-specific tasks like image/video editing @xcite or complex tax filings. integrating llms with external apis offers a solution, enabling real-time access to specialized resources and fostering innovations such as code interpreters @xcite . research on toolalpaca (tang * corresponding author, † equal contribution @xcite and @xcite demonstrates the potential of open-source llms in function-calling scenarios, extending their utility to iot, edge computing, and automated software development. enhancing llm integration with apis requires balancing large-scale model capabilities and efficiency. while large models like gpt-4 @xcite @xcite are powerful, they are computationally expensive for tasks using only a subset of apis. smaller, task-specific llms offer a cost-effective alternative @xcite @xcite but risk increased errors or "hallucinations" @xcite . precise output formatting is critical for software reliability @xcite , emphasizing the need for innovations that combine accuracy, efficiency, and reliability. to address these challenges, we propose a framework for training and inference tailored to task-specific llms. using a curated dataset of over 30,000 apis from rapid api hub @xcite , we employ curriculum learning @xcite to improve precision in selecting appropriate api functions. fine-tuning smaller models like codellama7b (roziere et al., 2023), google's gemma (gemma team, google deep-mind, 2023), and stable code 3b @xcite demonstrates superior performance over gpt-4 on specific benchmarks. the framework also supports deployment on resourceconstrained platforms such as mobile devices (team, 2023), ensuring broad applicability. to ensure output consistency, we introduce a conditional masking technique tailored for api function calls. unlike generic constrained decoding, this approach dynamically restricts token predictions to valid options based on the api schema, such as permissible parameter types and argument names. this guarantees syntactic and semantic correctness, significantly reducing errors while preserving inference speed. mathematical validation further demonstrates consistent improvements in accuracy, making this technique reliable for diverse real-world api interactions. in summary, this paper makes the following key contributions: these contributions collectively advance the field of automated software development by addressing critical inefficiencies in llm deployment for api interactions, providing open resources for the community, and setting a foundation for further research in task-specific llm optimization and application. ## methodology in this section, we outline our approach to dataset collection, preparation, and model development, detailing the steps taken to optimize the training process for api function calling tasks. we introduce the workflow designed to curate, format, and refine the dataset to ensure its suitability for effective model fine-tuning. furthermore, we describe the architecture and training process of our model, octopus, including the innovative techniques applied to enhance inference accuracy and efficiency. ## llm evaluation for function calling we evaluated the octopus model's ability to interpret and execute api function calls, comparing its performance to gpt-4 and gpt-3.5-turbo. the evaluation focused on function name recognition and parameter generation, with and without the use of conditional masking. the test set contains a vast diversity of apis in the real world. ## conclusion this study introduces a novel framework for training large language models on practical software apis and evaluates their performance in api calling tasks, surpassing gpt-4 in specific scenarios. our approach includes a refined dataset preparation methodology, leveraging negative sampling and curriculum learning to enhance model performance. additionally, we propose a conditional masking technique to address mismatched output formats, significantly improving accuracy and robustness in api function generation.