Cognitive Kernel: An Open-source Agent System towards Generalist Autopilots (2025)

Abstract: We introduce Cognitive Kernel, an open-source agent system towards the goal of generalist autopilots. Unlike copilot systems, which primarily rely on users to provide essential state information, autopilot systems complete tasks from start to finish independently. This requires the system to acquire the missing state information actively. Cognitive Kernel adopts a dynamic programming design where the central policy model (a fine-tuned LLM) could initiate an environment state perception task, essentially another agent task, as needed. The results demonstrate that Cognitive Kernel achieves better or comparable performance to other closed-source systems on core autopilot capabilities. Cognitive Kernel is fully dockerized, ensuring everyone can deploy it privately and securely. We open-source the system to encourage further research on LLM-driven autopilot systems

## introduction large language models (llms) have revolutionized the landscape of ai applications @xcite @xcite . systems like chatgpt 1 have significantly enhanced productivity in everyday tasks. however, these systems primarily function as "copilots," where users are required to manage the majority of the work, such as planning the overall workflow, posing the right questions, or refining the model's output as needed @xcite . to fully harness the capabilities of llms and reduce the burden of tedious, repetitive tasks, we shall shift from building "copilot" systems to "autopilot" systems that can independently complete tasks. for instance, while a copilot system might assist in drafting a template for an invitation email, an autopilot system should be capable of composing the entire email and sending it. at the same time, an autopilot system should be a general-purpose assistant capable of handling various user needs instead of being customized for a particular environment (e.g. an email agent). to achieve this goal, one should give the agent system the capability of proactively acquiring the essential environment information when needed rather than passively waiting for humans to provide through prompts (figure 1 (a) ). the system should also adaptively interact with the suitable environment rather then being constrained to a certain task-specific environment (figure 1 (b) ). following this design principle, we propose cognitive kernel, which has the freedom to interact with the real world to acquire missing information. as illustrated in figure 1(c ), cognitive kernel containts three conceptual components: the reasoning kernel, the perception kernel, and the memory kernel, corresponding to decision-making, state perception, and state storage, respectively. analogous to a turing machine @xcite , the reasoning kernel functions as the transition mapping mechanism, the perception kernel serves as the tool for reading current state information, and the memory kernel acts as the tape, recording past state information. at each step, the reasoning kernel will generate the next action, which might involve activating the perception kernel to gather missing state information or engaging the memory kernel to store or retrieve critical historical information. since such perception cannot be pre-defined, we treat the perception task as another fine-grained autopilot task in a dynamic programming paradigm. this autopilot agent design poses significant challenges to the central policy model of the system, as it would need to handle general chats with users, environment perception planning and decision-making within diverse environments. hence directly applying a fixed model leads to unsatisfactory performance. to address this issue, cognitive kernel employs a two-stage training process that enables the policy model to develop fundamental capabilities and then continuously learn from interaction feedback for further improvement after initial deployment. we evaluate the system's performance across three essential capabilities for an "autopilot" system: real-time information management (e.g., gathering data and completing tasks on the open web), private information management (e.g., processing and understanding local files), and long-term memory management (e.g., personalizing the system based on user interactions). by comparing our system with leading closed-source systems (e.g., chatgpt website, kimi website) and open-source agent systems utilizing different foundation models, we have made the following observations: (1) no single system consistently delivers the best performance across all tasks; each system has its strengths as well as inherent behavioral biases; (2) optimal performance is achieved through the deep integration of model and system design. cognitive kernel demonstrates significantly better results when paired with our specifically adapted model than an overall stronger model (gpt-4o). to summarize, cognitive kernel makes the fol-lowing contributions: (1) proactive state acquisition: our system introduces an innovative approach to proactively acquiring state information, enabling the possibility of evolving from a "copilot" to an "autopilot" system. ## system design in this section, we introduce the motivations and design principles of the three kernels in our cognitive kernel system. 3 in figure 2 , we demonstrate an example usage of our cognitive kernel system. ## applications in this section, we showcase another application scenario of cognitive kernel to illustrate how users might interact with the system. in this example from figure 10 , we see that the user first asked the system to search for recent papers about web agents and download the first one it found. cognitive kernel again opened a web browser and searched web agent papers. then it opened the first paper in the results and clicked the download button on the paper's arxiv page. after the paper was downloaded, the system returned the downloaded file's path. in the next turn, the user then asked how many times the keyword "html" is mentioned in the paper. then cognitive kernel leveraged its private information management ability to open the paper and count the occurrences of "html." then it returned the answer 5, which we verified to be correct by opening the paper and manually searching the keyword. ## conclusion in this paper, we presented cognitive kernel, a dockerized agent system towards the goal of general-purpose "autopilots", which has three main components: reasoning kernel for decisionmaking, perception kernel for state perceiving, and memory kernel for state management. we further reorganized these components into several dockers for easy and safe deployment. we evaluate cognitive kernel's ability to handle real-time information, private information and long-term histories. the results show that cognitive kernel achieves stronger or similar performance compared to other sota closed-source systems. we release our system framework, model weights, and findings to the community, hoping to inspire future research on the direction of generalist "autopilots" systems.