Language Agents: Foundations, Prospects, and Risks (2024)

Abstract: Language agents are autonomous agents, usually powered by large language models, that can follow language instructions to carry out diverse and complex tasks in real-world or simulated environments. It is one of the most heated discussion threads in AI and NLP at present with many proof-of-concept efforts, yet there lacks a systematic account of the conceptual definition, theoretical foundation, promising directions, and risks of language agents. This proposed tutorial aspires to fill this gap by providing a conceptual framework of language agents as well as giving a comprehensive discussion on important topic areas including tool augmentation, grounding, reasoning and planning, multi-agent systems, and rissk and societal impact. Language played a critical role in the evolution of biological intelligence, and now artificial intelligence may be following a similar evolutionary path. This is remarkable and concerning at the same time. We hope this tutorial will provide a timely framework to facilitate constructive discussion on this important emerging topic.

## introduction a heated discussion thread in ai and nlp is autonomous agents, usually powered by large language models (llms), that can follow language instructions to carry out diverse and complex tasks in real-world or simulated environments. there are numerous proof-of-concept efforts on such agents recently, including chatgpt plugins, 1 autogpt, 2 generative agents @xcite , just to name a few. the public is also showing an unprecedentedly high level of excitement. for example, autogpt has received 147k stars in just 4 months, making it the fastest growing repository in the github history, despite its experimental nature with many known and sometimes serious limitations. however, the concept of agent has been introduced into ai since its dawn. so what has changed recently? we argue that the most fundamental change is the capability of using language. contemporary ai agents use language as a vehicle for both thought and communication, a trait that was unique to humans. this dramatically expands the breadth and depth of the problems these agents can possibly tackle, autonomously. the capability of using language, bestowed by their llm foundations, allows these agents to 1) use a wide range of tools and reconcile their heterogeneous syntax and semantics @xcite @xcite @xcite , 2) operate in complex environments and ground to environmentspecific semantics @xcite @xcite @xcite , 3) conduct complex language-driven reasoning @xcite @xcite , and 4) form spontaneous multi-agent systems @xcite . therefore, to distinguish from the earlier ai agents, we suggest that these ai agents capable of using language for thought and communication should be called "language agents," for language being their most salient trait. language played a critical role in the evolution of biological intelligence, and now artificial intelligence may be following a similar evolutionary path. this is remarkable and concerning at the same time. despite the rapid progress, there has been a significant lack of systematic discussions regarding the conceptual definition, theoretical foundation, promising directions, and risks associated with language agents. this proposed tutorial endeavors to fill this gap by giving a comprehensive account of language agents based on both contemporary and classic ai research while drawing connections to cognitive science, neuroscience, and linguistics when appropriate. ## outline of tutorial content this cutting-edge tutorial will be half-day and cover a conceptual framework for language agents as well as important topic areas including tool augmentation, grounding, reasoning and planning, multi-agent systems, and risks and societal impact. ## other required information the proposed tutorial is considered a cutting-edge tutorial that gives a systematic account of the emerging topic of language agents. there is no prior tutorial at *cl conferences that has covered this topic. there are a few recent tutorials covering some related aspects of language agents, such as "acl'23: tutorial on complex reasoning over natural language" on reasoning, "acl'23: retrievalbased language models and applications" on retrieval augmentation, and "emnlp'23: mitigating societal harms in large language models" on societal considerations of llms. however, there lacks a comprehensive coverage on the foundations, prospects, and risks of language agents, a void this proposed tutorial aspires to fill. ## ethics statement language agents, with the ability of autonomously acting in the real world, pose significant potential ethical and safety risks. a main purpose of this proposed tutorial is to systematically define and analyze the unique capabilities and associated risks of language agents. we have a dedicated section on risks and societal impact, and we also cover related discussion in every other section when appropriate.