D roid C all: A Dataset for LLM -powered Android Intent Invocation (2025)

Abstract: The growing capabilities of large language models in natural language understanding significantly strengthen existing agentic systems. To power performant on-device mobile agents for better data privacy, we introduce DroidCall, the first training and testing dataset for accurate Android Intent invocation. With a highly flexible and reusable data generation pipeline, we constructed 10k samples in DroidCall. Given a task instruction in natural language, small language models such as Qwen2.5-3B and Gemma2-2B fine-tuned with DroidCall can approach or even surpass the capabilities of GPT-4o for accurate Android intent invocation. We also provide an end-to-end Android app equipped with these fine-tuned models to demonstrate the Android intent invocation process. The code and dataset are available at https://github.com/UbiquitousLearning/DroidCall

## introduction the advent of large language models (llms) revolutionizes natural language processing, enabling machines to understand and generate human-like language with unprecedented accuracy. in the realm of mobile computing, this advancement presents a significant opportunity for developing intelligent mobile agents @xcite @xcite . specifically, these agents can leverage the rich ecosystem of built-in intents (google, 2024a) provided by both the operating system and thirdparty applications on android devices. these intents serve as a fundamental mechanism for interapp communication and function invocation, such as sending messages, making phone calls, or triggering specific app features. by harnessing llms, mobile agents can interpret diverse and complex user instructions, seamlessly mapping them to the appropriate intents, and therefore automating user interaction with mobile devices. figure 1 : small language models fine-tuned with droidcall have the capability to assist users in completing common tasks such as adding events to the calendar. on-device llms are necessary for building mobile agents due to privacy and latency constraints @xcite @xcite @xcite . since user data are processed locally, sensitive information remains on devices, thereby mitigating risks associated with data transmission over networks. moreover, on-device inference eliminates the need for constant internet connectivity. various ondevice llm inference optimizations significantly reduce response time @xcite @xcite , leading to a more responsive and fluid user experience. however, our investigations reveal a critical challenge: existing device-affordable llms lack the capability of accurate intent invocation. for example, llama3.2-1b @xcite only succeeds in 31.5% and 60.5% of the tasks in zero-shot and few-shot scenarios, respectively. this limitation is not due to inherent deficiencies in the models themselves but stems from the absence of specialized datasets tailored for this purpose. existing llms are typically trained on broad datasets that do not encompass the specific language patterns and contextual nuances required for accurate intent invocation. to address this gap, we introduce droidcall, the first open-sourced, high-quality dataset designed for fine-tuning llms for accurate intent invocation on android devices, along with a flexible and reusable data generation pipeline. droidcall comprises an extensive collection of user instructions paired with their corresponding intents, covering a wide array of functionalities across the system and third-party apps while the data generation pipeline automatically generates, validates, and deduplicates data to ensure accuracy and diversity. unlike existing methods @xcite @xcite , our approach eliminates the need for manually written seed data, significantly reducing labor. ## droidcall dataset and workflow this section outlines the droidcall framework, detailing its three key phases for building android intent invocation capabilities: function predefinition, data generation, and model fine-tuning & evaluation, as illustrated in figure 2 . our data generation method requires minimal human supervision and is easily extensible. we conclude with an end-to-end demonstration of device control using fine-tuned llms. ## experiments we first explore the impact of prompt designs on model fine-tuning, identifying an optimal format. we then demonstrate droidcall's superior performance over general function-calling datasets for android intent invocation. next, we analyze the contribution of different droidcall data types through an ablation study. finally, we present results showcasing the effectiveness of models finetuned with droidcall. metrics. we introduce two metrics to evaluate function-calling performance: accuracy and soft accuracy. evaluates partially correct function calls by calculating the proportion of accurately predicted parameters, averaged across all calls: we evaluate slms using the 200 test entries from droidcall. following the input format used during training ( §3.3), the prompts for evaluation include descriptions of the relevant functions required for each task. this setup allows us to isolate and measure the slm's core ability to map user queries to correct function calls, decoupling the evaluation from the function retrieval process required in a full end-to-end system (demonstrated in §3.4). in § 3.3, we described the model input components: user query, available function descriptions, and task instructions, with the output being a specific representation for calling a function. while the user query is user-provided, we designed the remaining components to evaluate their impact on fine-tuning performance. json. uses json for available function descriptions and function call representations due to its simplicity. we fine-tuned qwen2.5-1.5b-instruct using four prompt formats. figure 5(a) shows accuracy. the code_short format achieved comparable results to json with significantly fewer tokens ( ## conclusion in this paper, we introduce droidcall, a novel dataset specifically engineered to enhance the android intent invocation capabilities of llms. our approach diverged from conventional cloud-based models, focusing instead on on-device deployment to address privacy concerns inherent in mobile environments. in our work, we (1) build a highly customizable and reusable data generation pipeline, (2) construct droidcall, a first-of-its-kind opensourced dataset for android intent invocation based on the pipeline, (3) fine-tune a series of models tailored for edge devices, enabling them to approach or even surpass the performance of gpt-4o in the specific task of intent invocation and (4) implement an end-to-end demo with mllm. our work demonstrates the potential applications of small models on the edge. we have open-sourced all the code of the data generation, fine-tuning, and evaluation.