A gent S tore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant (2025)

Abstract: Digital agents capable of automating complex computer tasks have attracted considerable attention. However, existing agent methods exhibit deficiencies in their generalization and specialization capabilities, especially in handling open-ended computer tasks in real-world environments. Inspired by the rich functionality of the App store, we present AgentStore, a scalable platform designed to dynamically integrate heterogeneous agents for automating computer tasks. AgentStore allows the system to continuously enrich its capabilities and adapt to rapidly evolving operating systems. Additionally, we propose a novel core MetaAgent with the AgentToken strategy to efficiently manage diverse agents and utilize their specialized and generalist abilities for both domain-specific and system-wide tasks. Extensive experiments on three interactive real-world benchmarks demonstrate that AgentStore significantly expands the capability boundaries of agent systems in both generalization and specialization, underscoring its potential for developing the specialized generalist computer assistant.

## introduction the continual evolution of computer operating systems (os), along with proliferating applications, has transformed how people work and live. this transformation goes beyond daily life like shopping and gaming, encompassing professional works such as writing in office or editing in photoshop. however, this increased functionality comes with a steep learning curve, often burdening users. as a result, autonomous computer assistants-once limited to fiction like jarvis in iron man or moss in wandering earth-have become a concrete pursuit, attracting great interest from researchers. equal corresponding author. 1 the concept of "specialized generalist" refers to an ai system that excels in specific tasks while still maintaining broad general capabilities @xcite . advancements in multimodal large language models (mllms) @xcite , are gradually turning this vision into reality. however, real-world os environments encompass a diverse array of open-ended computer tasks, each with inherent requirements for capabilities across multi-dimensions @xcite , posing substantial challenges to existing methods. specifically, "task_1" in figure 1 illustrates that many computer tasks necessitate specific knowledge and operations. in such scenarios, existing generalist agents @xcite often underperform due to their lack of these specialized abilities. conversely, specialized agents, despite excelling at specific tasks within single domains like tabular data processing @xcite or web browsing @xcite , struggle to perform when confronted with more integrated, system-wide tasks like "task_2" in figure 1 . this heterogeneous demand for capabilities presents a challenge for existing single generalist or specialized agents. we attribute this dilemma to overlooking a key factor behind the success of modern operating systems: app store foot_0 which continuously expands the range of functionalities beyond the core os itself. correspondingly, we argue that specialized generalist computer agents should possess this characteristic, evolving to grow heterogeneous abilities and autonomously handle an increasingly diverse range of tasks. to substantiate this, we propose agentstore, a flexible and scalable platform for dynamically integrating various heterogeneous agents to independently or collaboratively automate os tasks (illustrated on the right in figure 1 ). we develop a prototype of agentstore, establishing an integration protocol and implementing over 20 computer os agents and 10 mobile os agents each integrated agent in agentstore is denoted as a learnable token embedding in metaagent's architecture like a word token embedding. during inference, metaagent activates the corresponding agent to execute the task when an agent token is predicted. innovatively, we enhance this approach by shifting from single-token @xcite to multi-token prediction, allowing metaagent to predict and coordinate multiple agents for collaborative task execution. additionally, we propose an automated process with self-instruct for tuning agenttoken without relying on manual data. we validate the effectiveness of agentstore across three real-world dynamic benchmarks involving both computer and mobile os platforms. on the highly challenging osworld benchmark, agentstore achieves a success rate of 23.85%, more than doubling the performance of the previous generalist agent (11.21%) @xcite . on a novel benchmark osworld-multi for evaluating agent collaboration, agentstore significantly improves the execution success rate from 6.93% to 22.77%, and achieves strong performance across all metrics. these results highlight the importance of scalable agent integration in expanding the system's capabilities. similar outcomes are ob-served when evaluating agentstore in mobile environments. additionally, we also demonstrate the advantage of agenttoken in comparison to other strategies, highlighting its efficiency in training and its effectiveness in dynamic management. ## agentstore as illustrated in figure 2 , agentstore consists of two main components: agentpool and metaagent. the agentpool (in section 3.1) stores featurespecific agents with distinct functionalities and defines the integration protocol for adding new agents. the metaagent (in section 3.2 and 3.3) selects the most suitable agent(s) from agentpool to independently or collaboratively complete tasks. ## experiments to assess the effectiveness and versatility of agentstore, we conduct comprehensive experiments across various tasks. we demonstrate its outstanding performance on three real-world os benchmarks (in section 4.1), followed by an indepth analysis of agent integration and agenttoken (in section 4.2 and 4.3). we also provide running cases for qualitative analysis (in section 4.4). ## conclusion in this paper, we introduce agentstore, a flexible and scalable platform for dynamically integrating heterogeneous agents to independently or collaboratively complete os tasks. furthermore, we propose metaagent with the agenttoken strategy to achieve efficient management of the growing number of agents. comprehensive quantitative analysis and qualitative results show that agentstore expands the capabilities of existing agent systems in both generalization and specialization. we believe that as basic agi models continue to evolve, agentstore will integrate more powerful agents, progressively advancing toward the vision of building the specialized generalist computer assistant. this research focuses on building a scalable platform to integrate heterogeneous agents dynamically. the data datasets or benchmarks we employed are properly cited. there are no discrimination, bias, or fairness issues that need to be declared in this paper. further, the outputs are not expected to be potentially harmful. to ensure reproducibility, we provide experimental details in section 4 and their corresponding appendices. all source code will be made public.