CAMPHOR : Collaborative Agents for Multi-input Planning and High-Order Reasoning On Device (2025)

Abstract: While server-side Large Language Models (LLMs) demonstrate proficiency in function calling and complex reasoning, deploying Small Language Models (SLMs) directly on devices brings opportunities to improve latency and privacy but also introduces unique challenges for accuracy and memory. We introduce CAMPHOR, an innovative on-device SLM multi-agent framework designed to handle multiple user inputs and reason over personal context locally, ensuring privacy is maintained. CAMPHOR employs a hierarchical architecture where a high-order reasoning agent decomposes complex tasks and coordinates expert agents responsible for personal context retrieval, tool interaction, and dynamic plan generation. By implementing parameter sharing across agents and leveraging prompt compression, we significantly reduce model size, latency, and memory usage. To validate our approach, we present a novel dataset capturing multi-agent task trajectories centered on personalized mobile assistant use-cases. Our experiments reveal that fine-tuned SLM agents not only surpass closed-source LLMs in task completion F1 by ~35% but also eliminate the need for server-device communication, all while enhancing privacy.

## introduction server-side large language models (llms) are powerful semantic parsers that interpret user intent and map queries to executable function calls. to ground a query within a personal environment such as an open toolbox, retrieval-augmented generation (rag) @xcite can be adopted to pre-fill the llm prompt with external knowledge relevant to the user query, such as top-k most relevant tools. an orthogonal strategy is long-context language modeling @xcite , which pre-loads the prompt with all available external knowledge, taking advantage of a larger context window up to 128k tokens @xcite . however, a server-side llm is not optimal for a mobile assistant due to privacy and latency concerns. privacy. user queries to mobile assistants are often ambiguous, making it crucial to ground them in personal information, such as contacts, installed tools, and past activities. while the assistant needs access to personal data to improve understanding, it must also prioritize user privacy by keeping sensitive information on the device. even private cloud solutions are not ideal for this, as they commonly avoid storing user-specific data, which prevents kv caching in multi-turn dialogues @xcite . latency. in addition to the limited flexibility of kv caches due to privacy concerns, server-side llms introduce extra latency between understanding (which occurs on the server) and execution (which happens on the user's device). this latency can degrade the user experience, particularly for solutions requiring multiple server-device round trips. for instance, the react framework @xcite , which breaks down the understanding task into multiple steps and reasons over intermediate execution results, is hindered by this added latency. a more effective solution to address latency and privacy concerns is to deploy a small language model (slm) on-device, allowing it to process personal data and interpret user queries locally. however, slms introduce new challenges related to accuracy due to tighter budgets on prompts and kv caches. while a rag approach is scalable across personal databases, the retriever is a separate model which gates the performance of the language model since it is impossible to achieve a perfect recall given the fixed prompt budget @xcite . the problem becomes worse for compositional queries which are naturally harder for retrieval. consider the following example: can you show me the cheapest flight options to barcelona next month and add it to my calendar? also, let my travel buddy know about our trip plan. the query is both compositional and includes a personal entity (my travel buddy). successfully retrieving the appropriate tool requires task decomposition, which depends on the granularity of the available toolbox (i.e., the tools present on the user's device). while long-context modeling allows the language model to consider all relevant tools and personal entities during query parsing, it also rapidly exhausts the prompt budget, increases kv cache overhead, and introduces a "needle-in-ahaystack" challenge @xcite for the slm. we argue that a key to successful on-device understanding is an agentic approach that decomposes the understanding task in functional space, proactively seeks personal information and runtime feedback before generating the final function call. specifically, we propose camphor, an slmbased collaborative agent understanding system for grounded query parsing. camphor features a hierarchical architecture, with a high-order reasoning agent that incrementally solves a query by orchestrating sub-agents. each sub-agent is responsible for a unique functionality, such as retrieving personal entities, fetching latest user activities, and calling task-completion apis. we apply tool compression techniques that enable the slm to reason over device-specific toolboxes while satisfying prompt and memory budgets. overall, camphor is designed to offer optimal accuracy and speed, while protecting user data on device. we release the camphor dataset foot_0 , which consists of multi-agent execution trajectories focused on mobile assistant use cases (as shown in figure 1 ). the dataset is unique in how it simulates user device states and ground query understanding to personal information. we fine-tune the slmbased camphor agent on this dataset, and prove that fine-tuned slms can outperform closed-source llms in terms of task completion metrics, while eliminating server-device communication costs and safeguarding user privacy. @xcite , when fine-tuned for specific tasks, can outperform prompting large language models (llms). in particular for the function calling task, the octopus series @xcite has achieved remarkable accuracy, exceeding 97% for function calling on device. most related to our work is the on-device octo-planner @xcite , which breaks down a query into multiple subqueries for function call generation. however, we argue that query decomposition in natural language space is an unconstrained optimization problem, as the granularity of decomposition depends on the available toolset. multi-agent planning is the process where multiple agents, each with unique capabilities, knowledge, and objectives, work together towards shared or interrelated goals. the rise of large language models (llms) has significantly advanced the development of multi-agent planning, as tasks for each agent can potentially be solved through prompts. agentic frameworks like re-act @xcite , reflexion @xcite , lats (zhou et al.), swiftsage (lin et al., 2024), and autoact @xcite continuously prompt llms to reflect on and critique their past ac-tions, sometimes incorporating additional external information, such as environmental observations and feedback. in this work we focus on multi-agent that solves user queries while understanding user data on a device. examples of such data includes past user actions, personal entities and installed toolsets @xcite . retrieval augmented generation and longcontext language models are two orthogonal approaches to ground a fine-tuned lm with external data sources, which in this work include the dynamic set of personal entities and tools @xcite . a standard workflow of rag includes possibly a query generation step (or a query decomposition step for compositional utterances) @xcite followed by sparse and/or dense retrieval. the retrieval model is commonly a separate set of parameters which can be trained either separately or jointly with the llm. a major limitation of rag is that a sub-optimal retrieval model will gate the performance of the llm which has access to more contextual information. in contrast, long-context llms allow for the direct incorporation of more external data into the prompt @xcite @xcite . however, this comes at the cost of increased size of prompt and kv caches, making it impractical for small language models (slms) and on-device deployment. prompt compression is an optimization to reduce the number of prompt tokens at least at the inference time. we adopt the technique to enable slms to retrieve directly from a dynamic toolbox. related to this work are the work of gist tokens @xcite , parallel context encoder @xcite , and squid which compresses a piece of long text into a single embedding @xcite . these approaches differ in terms of how the compressed embedding is learned and incorporated with the base llm, as either prompt tokens or late fusions in the attention layer. ## camphor dataset a central focus of camphor is personalized planning and query understanding on device. however, existing function calling datasets @xcite only provide task completion annotations for user queries but not incorporating personal knowledge for understanding. on the other hand, there exist a few datasets on agent planning but they largely focus on mathematics @xcite @xcite and common sense reasoning @xcite instead of query parsing. to this end, we created the camphor dataset by annotating each query with a trajectory of function calls that demonstrate how a multi-agent system proactively fetches personal information to solve a user query by breaking down the understanding task into smaller actions. the dataset is developed by assigning a personal device state to each query, which includes a randomly sampled history of user activities, as well as the personal entities and tools available on the device. each query in the dataset is generated by gpt-4o based on a device state and a set of global function definitions. the gpt-4o is also instructed to annotate the query in a multi-step fashion. the execution results are fetched from the device state for each function calling, which are then used to guide the next step of annotation. the final solution path is reviewed and verified with human oversight. overall, the camphor dataset contains 3,410 queries, which are split into 2,728 for training and 682 for test. the dataset is flattened, resulting in 35,444 prompt-completion pairs for slm fine-tuning, with an average of 10.39 pairs per query. ## experiments we consider two slm candidates for fine-tuning the camphor agents: phi-3.5 and gemma-2. the sequence of prompt and completion pairs associated with each query is obtained by unrolling the ground truth trajectories in the dataset. fine-tuning details are provided in appendix a. ## conclusion this work introduces camphor, a collaborative, slm-based agent framework designed for personalized query parsing on user devices. camphor proactively retrieves on-device information and decomposes the understanding tasks into multiple steps of function calls. our results show that a finetuned slm outperforms instruction-based llms in this task. by employing advanced prompt compression techniques, camphor strikes an optimal balance between accuracy and efficiency, while safeguarding user data directly on the device.