OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
Abstract
Autonomous interaction with the computer has been a longstanding challenge with great potential, and the recent proliferation of large language models (LLMs) has markedly accelerated progress in building digital agents. However, most of these agents are designed to interact with a narrow domain, such as a specific software or website. This narrow focus constrains their applicability for general computer tasks. To this end, we introduce OS-Copilot, a framework to build generalist agents capable of interfacing with comprehensive elements in an operating system (OS), including the web, code terminals, files, multimedia, and various third-party applications. We use OS-Copilot to create FRIDAY, a self-improving embodied agent for automating general computer tasks. On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods by 35%, showcasing strong generalization to unseen applications via accumulated skills from previous tasks. We also present numerical and quantitative evidence that FRIDAY learns to control and self-improve on Excel and Powerpoint with minimal supervision. Our OS-Copilot framework and empirical findings provide infrastructure and insights for future research toward more capable and general-purpose computer agents.
Community
I wonder how stable and consistent it is!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Experiential Co-Learning of Software-Developing Agents (2023)
- WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024)
- Small LLMs Are Weak Tool Learners: A Multi-LLM Agent (2024)
- AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (2024)
- GitAgent: Facilitating Autonomous Agent with GitHub by Tool Extension (2023)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Amazing work! I note several great ideas:
- Task planning as an acyclic graph is more flexible than other frameworks like ReAct.
- When the code execution fail, retry the same task but let the agent refine its call. This probably gives the system more perseverance.
- Genius idea: let the model create tools and later retrieve them when needed with RAG if they have been successful. This is a game-changer for versatility!
Thank you @gregmialz for pointing me to this! 🤗
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper