T iny A gent: Function Calling at the Edge (2024)

Abstract: Recent large language models (LLMs) have enabled the development of advanced agentic systems that can integrate various tools and APIs to fulfill user queries through function calling. However, the deployment of these LLMs on the edge has not been explored since they typically require cloud-based infrastructure due to their substantial model size and computational demands. To this end, we present TinyAgent, an end-to-end framework for training and deploying task-specific small language model agents capable of function calling for driving agentic systems at the edge. We first show how to enable accurate function calling for open-source models via the LLMCompiler framework. We then systematically curate a high-quality dataset for function calling, which we use to fine-tune two small language models, TinyAgent-1.1B and 7B. For efficient inference, we introduce a novel tool retrieval method to reduce the input prompt length and utilize quantization to further accelerate the inference speed. As a driving application, we demonstrate a local Siri-like system for Apple’s MacBook that can execute user commands through text or voice input. Our results show that our models can achieve, and even surpass, the function-calling capabilities of larger models like GPT-4-Turbo, while being fully deployed at the edge. We open-source our [dataset, models, and installable package](https://github.com/SqueezeAILab/TinyAgent) and provide a [demo video](https://www.youtube.com/watch?v=0GvaGL9IDpQ) for our MacBook assistant agent.

## introduction the ability of llms to execute commands through plain language (e.g. english) has enabled agentic systems that can complete a user query by orchestrating the right set of tools (e.g. tool-former @xcite , @xcite ). this, along with the recent multi-modal *equal contribution 1 https://github.com/squeezeailab/tinyagent 2 https://www.youtube.com/watch?@xmath0, 2024) or gemini-1.5 @xcite , has expanded the realm of possibilities with ai agents. however, the large model size and computational requirements of these models often requires their inference to be performed on the cloud. this can create several challenges for their widespread adoption. first, uploading data such as video, audio, or text documents to a third-party vendor on the cloud, can result in privacy issues. second, this requires cloud/wi-fi connectivity which is not always possible. for instance, a robot deployed in the real world may not always have a stable connection. besides that, latency could also be an issue as uploading large amounts of data to the cloud and waiting for the response could slow down response time, resulting in unacceptable time-to-solution. these challenges could be solved if we deploy the llm models locally at the edge. current llms like gpt-4o (openai, 2024) or gemini-1.5 @xcite are too large for local deployment. one contributing factor is that a lot of the model size ends up memorizing general information about the world into its parametric memory which may not be necessary for a specialized downstream application. for instance, if you ask a general factual question to these models like a historical event or well-known figures, they can produce the results using their parametric memory, even without having additional context in their prompt. this implicit memorization of training data into the parametric memory might be correlated with "emergent" phenomena in llms such as in-context learning and complex reasoning, which has been the driving force behind scaling the model size. ## putting it all together we provide a demo video of the final tinyagent-1.1b model deployed on a macbook pro m3 3 , which can be downloaded and tested on mac from the link foot_2 . it not only runs all of the model inference locally on your computer, but it also allows you to provide commands through audio. we process the audio locally as well using the whisper-v3 @xcite model from openai deployed locally using the whisper.cpp framework. the greatest surprise for us was that the accuracy of the 1.1b model exceeds that of gpt-4-turbo, and is markedly fast while deployed locally and privately on-device. ## conclusions to summarize, we introduced tinyagent and showed that it is indeed possible to train a small language model and use it to power a semantic system that processes user queries. in particular, we considered a siri-like assistant for mac as a driving application. the key components for enabling it is to (i) teach off-the-shelf slms to perform function calling through llmcompiler framework, (ii) curate high quality function calling data for the task at hand, (iii) fine-tune the off-the-shelf model on the generated data, and (iv) enable efficient deployment by optimizing the prompt size through only retrieving the necessary tools based on the user query through tool rag, as well as quantized model deployment to reduce inference resource consumption. after these steps, our final models achieved 80.06% and 84.95% for the tinyagent-1.1.b and 7b models which exceed gpt-4-turbo's success rate of 79.08% on this task.