Towards large language model-based personal agents in the enterprise: Current trends and open problems (2023)

Abstract: There is an emerging trend to use large language models (LLMs) to reason about complex goals and orchestrate a set of pluggable tools or APIs to accomplish a goal. This functionality could, among other use cases, be used to build personal assistants for knowledge workers. While there are impressive demos of LLMs being used as autonomous agents or for tool composition, these solutions are not ready mission-critical enterprise settings. For example, they are brittle to input changes, and can produce inconsistent results for the same inputs. These use cases have many open problems in an exciting area of NLP research, such as trust and explainability, consistency and reproducibility, adherence to guardrails and policies, best practices for composable tool design, and the need for new metrics and benchmarks. This vision paper illustrates some examples of LLM-based autonomous agents that reason and compose tools, highlights cases where they fail, surveys some of the recent efforts in this space, and lays out the research challenges to make these solutions viable for enterprises.

## introduction the emergence of chatgpt has put large language models (llms) @xcite under the microscope to determine what they can and cannot do @xcite . with llms outperforming state-of-the-art approaches for traditional areas of natural language processing (nlp) research @xcite , the focus shifted to other ways that llms can be applied. the last few months, for example, have brought on a flurry of activity around llms as an orchestrator or brain of autonomous agents tasked with completing high level goals @xcite @xcite . autonomous agents perceive the world they operate in, reason about the events within it and their goal, decompose this goal into sub-problems, and task list llm-based agent external tools knowledge and context high level goal (1) user goal added to task list (2,4) task prioritization: based on goals, tasks, knowledge, context, and available tools (3) reasoning on tasks: decompose tasks to sub-tasks (5) tool selection and execution: invoke tools (6) tool observation: interprets tool output (7) reasoning on output: update tasks based on tool output perform a sequence of actions (equivalent to composing external tools and apis to carry out tasks) to achieve a new state in this world @xcite . figure 1 depicts the iterative pattern that some llm-based autonomous agents are applying @xcite @xcite . philosophically, it is perhaps not surprising that language models are being used for reasoning and carrying out complex goals. for example, the sapir-whorf hypothesis @xcite suggests that language influences thought. better language models then should be capable of more sophisticated thinking, including reasoning and planning. another connection is to the modes of thinking in humans: system 1 for instinctive thought processes, and system 2 for more deliberative thought @xcite . one can consider nlp tasks such as entity recognition as belonging to the system 1 bucket, whereas tasks where llms are used to reason and plan belong to the system 2 bucket. as language models become more powerful, perhaps, they acquire system 2 abilities. in enterprise settings, autonomous agents often take the form of personal assistants capable of conversing in natural language and executing actions that alter the state of the world (kephart, 2021; @xcite . among other functionality, these assistants carry out task-oriented dialog, further coupling language and reasoning. in addition to performing their functions accurately, enterprise-ready technology must be explainable and reliable. they must also adhere to strict regulations and governance, especially around concerns such as privacy, bias, and auditability. we notice, however, that beyond impressive demos, these llm orchestrators are brittle, fail in unpredictable ways, and are not consistent which is not ideal for enterprise applications. in this vision paper, we discuss the readiness of llm-based autonomous agents for enterprise applications. first, we survey both anecdotal and academic works that have implemented llm-based agents. then, we show empirical evidence of llms as tool composers in an enterprise travel use case. we rely on both analyses to define a set of challenges that must be addressed to make llms resilient enough to use as an orchestrator for mission-critical enterprise use cases. ## anecdotal uses of llm-based agents chatgpt created a lot of excitement around building llm-based autonomous agents, where researchers and non-researchers alike were inspired to create demos, applications, and services using llms. in this section, we review anecdotal accounts of llm-based autonomous agents and tool composers. they help stretch the imagination of what is possible, but do not perform experiments to stress test their solutions and understand the strengths and limitations of llms in this role. ## a review of llm-based agents in the previous section, we provided evidence of the feasibility of llm-based agents from nonacademic sources such as blogs, twitter posts, github repositories, and demos. while these generated excitement around llms outside of the research community, this evidence does not allow us to assess the true capabilities and limitations of llms. next, we survey published literature that more systematically analyzes llms in domainsagnostic and domain-specific settings. these papers consider a spectrum of actions from no sideeffects to irreversible as illustrated in figure 3 . domain-agnostic autonomous agents recent papers extend the use of llms beyond simple tasks such as summarization or entity extraction to handle complex reasoning and question answering. this includes augmenting llms with the ability to query the internet @xcite or to use of different types of tools (e.g. calculators) or access external information (e.g. google, or wikipedia search) @xcite . when @xcite leveraged llms to decompose complex tasks into sequence of actions that have no side effects, taskmatrix.ai and react present a vision that utilizes gpt-4 to connect any apis to complete tasks @xcite @xcite focused on plug and play when leveraging llms for tool composition. finally, @xcite ) created a benchmark with 500+ apis to evaluate the effectiveness of llms as tool composers. as depicted in figure 1 , an evolution of these agents adds long term memory and task prioritization capabilities. an llm decomposes high level goals to smaller tasks, a vector store is typically used for long term memory to recall outputs from long task lists, and an llm is used to iteratively reprioritize existing tasks and create new tasks based on the progress towards the goal @xcite @xcite . robotic agents due to their emergent behaviors @xcite , llms have been used to make goal-driven decisions. these approaches mainly rely on llms to accomplish tasks in the physical or virtual world using internet-of-things devices and robots @xcite @xcite . @xcite evaluate whether llms contain information necessary to accomplish goals without additional training or grounding. followup efforts on grounding primitive actions include text2motion @xcite , autotamp @xcite and saycan @xcite to guide llmbased task and motion planning. tidybot @xcite uses llms to infer generalized user preferences that are applicable to future interactions. llm-based interactive agents mimicked human behavior in @xcite . other domains dialog systems with the goals of performing tasks such as finding restaurants or reserving hotels have been addressed in the nlp community @xcite . for the banking sector, @xcite ) trained a finance specific llm. in the medical domain, @xcite tested chatgpt for clinical question-answering. @xcite ) created a public health intervention system that can chat with patients and notify health care professionals if intervention was necessary. in the area of online games, voyager @xcite uses gpt-4 to build an agent to continuously explore a minecraft world with the goal of making novel discoveries. it uses a catalog of skills or tools, and can also learn new skills. ## use case: chatbot for enterprise travel the integration of apis with llms @xcite has raised the question of whether llms can act as task-oriented chatbots, particularly on complex tasks that require the execution of a sequence of apis. automating tasks in enterprise settings is particularly difficult since each company has custom processes in place and often little data to train or fine-tune an llm to their peculiarities. ## research challenges we now draw on the limitations observed from anecdotal applications of llm-based agents (section 2), experimental evaluation of an enterprise travel case study (section 4), and our own experiences developing enterprise application platforms foot_0 to define a set of research challenges that need to be solved before llm-based agents can be applied to mission-critical enterprise use cases. ## final thoughts and next steps enterprises are on a continuous quest to optimize repetitive and tedious work. advancements in llms have broadened imaginations on what knowledge work can be automated, and llmbased autonomous agents can be the vehicle to deliver incredible productivity gains. attempts in the community to build agents using even state-of-the-art llms, as well as our case study using llms for an enterprise travel use case, reveal how inadequate these solutions are for mission-critical enterprise use cases. we outline several research challenges that the nlp and other research communities can investigate. while we describe individual challenges, such as robustness, multi-modality, and tool composition, the community must take a holistic view of the problem, addressing not just llm advances, but the entire software lifecycle including tool authoring, and debugging. defining clear use cases and benchmarks will be important steps. on the survey and challenges contributions of this work: our coverage of the literature is incomplete due to the extraordinary fast pace and sheer volume of work posted on arxiv, blogs, etc. furthermore, due to this rapid pace, we do cite a large number of non-peer-reviewed work. our discussion of existing challenges and how to resolve them is also colored by our experience in industry and may not include some more theoretical challenges that the community should also solve in conjunction with the practical challenges. on the experimental contributions of this work: our datasets are small in size compared to what may be commonly used, their labeling is noisy due to the pseudo-labeling approach we adopted and may not be very realistic due to the additional processing we performed to get them to the format we required. furthermore, the metrics we calculated do not measure all characteristics that we may want to evaluate. also, since we only used one ground truth to calculate the metrics, this may result in less accurate values (e.g., bleu becomes more accurate as more references are used). finally, the experimental analysis is not comprehensive, missing some ablation studies and other experiments that could help answer additional questions on the performance on llms on task-oriented dialog tasks.