REST ful-Llama: Connecting User Queries to REST ful API s (2024)

Abstract: Recent advancements in Large Language Models (LLMs) have showcased exceptional performance in zero-shot learning and reasoning tasks. However, integrating these models with external tools - a crucial need for real-world applications - remains a significant challenge. We propose RESTful-Llama, a novel framework designed to enable Llama 3.1 to transform natural language instructions into effective RESTful API calls. To enhance the fine-tuning process, we introduce DOC_Mine, a method to generate fine-tuning datasets from public API documentation. RESTful-Llama distinguishes itself by enabling open-source LLMs to efficiently interact with and adapt to any REST API system. Experiments demonstrate a 31.9% improvement in robustness and a 2.33x increase in efficiency compared to existing methods.

## introduction large language models (llms) have made significant strides in natural language processing (nlp) and various interdisciplinary domains in recent years. they exhibit the ability to engage in humanlike conversations and demonstrate the potential to integrate with external tools, such as search engines and productivity software @xcite -an essential feature for real-world applications. given the widespread use of these tools in daily activities, such integration is critical to meeting end-user needs and enhancing their interaction with technology. building on this potential, a promising area of research seeks to incorporate llms with multimodal tools. intelligent planners like visual chat-gpt @xcite and hugginggpt @xcite utilize pre-defined templates to generate instructions executable by various foundation models. while these strategies have shown impressive results, they are typically confined to a limited selection of specially designed tools or models, making them difficult to adapt or extend to other systems. moreover, the reliance on proprietary llms, such as chatgpt (openai, 2024a), raises concerns in the industry about potential data breaches. rather than focusing exclusively on a limited set of external tools, recent research efforts focus on improving llm generalization across a wider range of tasks. for example, react @xcite enables llms to interact with external environments such as alfworld @xcite and wikipedia to tackle general tasks. @xcite facilitates machine learning-related api calls through platforms like torchhub @xcite . despite their broader scope, these approaches often face limitations in task resolution success rates, which impedes their practical deployment in real-world scenarios. on the other hand, these methods struggle with scenarios where flexibility across diverse api systems is required, such as when integrating llms with dynamic, real-world api systems that evolve over time. to overcome the limitations of previous methods, this study introduces a novel method, restful-llama, which empowers open-source llms to translate user natural language queries into effective restful api calls for real-world applications. in summary, the main contributions of the study are as follows: ## dataset creation and fine-tuning given the widespread use and availability of rest api documentation, creating a diverse and highquality dataset for fine-tuning open-source llms is essential. in this section, we outline the steps involved in dataset creation and fine-tuning. as illustrated in figure 1 , the procedure includes: (i) using the doc_mine approach to generate the dataset from public api documentation, (ii) cleaning and filtering the generated data, and (iii) the supervised fine-tuning (sft) process. ## the restful-llama framework as illustrated in figure 2 , restful-llama consists of four main steps: (1) initialization with api documentation ingestion, (2) extracting essential information from the user query, (3) identifying relevant api documentation stored in the vector store, and (4) translating the user query into a python script using the fine-tuned llama 3.1-8b model, as described in section 3.3. the complete workflow is summarized in algorithm 1. ## experiment in this section, we evaluate the effectiveness of our approach, restful-llama, compared to the baseline react method. we also conduct case studies to investigate the impact of varying the k value in vector store retrieval. ## industrial application in this section, we briefly describe how restful-llama is adapted and deployed in the real world. as shown in figure 3 , restful-llama is integrated into a fire department system to assist firefighters in their operations. this integration enables individuals with no technical background and minimal training to effectively use the system. users can submit queries, which restful-llama translates into system commands, displaying the results back to the users. ## conclusion in this paper, we introduce restful-llama, a novel framework that bridges the gap between natural language processing and restful api operations for real-world applications. specifically, we finetune the llama 3.1 model to better align with the restful-llama framework, which subsequently generates python scripts that invoke the desired rest apis according to user queries. our experimental results demonstrate that restful-llama outperforms existing methods, improving both robustness and efficiency in api interactions.