Middleware for LLM s: Tools Are Instrumental for Language Agents in Complex Environments (2024)

Abstract: The applications of large language models (LLMs) have expanded well beyond the confines of text processing, signaling a new era where LLMs are envisioned as generalist agents capable of operating within complex environments. These environments are often highly expansive, making it impossible for the LLM to process them within its short-term memory. Motivated by recent research on extending the capabilities of LLMs with tools, we seek to investigate the intriguing potential of tools to augment LLMs in handling such complexity by introducing a novel class of tools, termed *middleware*, to aid in the proactive exploration within these massive environments. Such specialized tools can serve as a middleware layer shielding the LLM from environmental complexity. In two representative complex environments—knowledge bases (KBs) and databases—we demonstrate the significant potential of augmenting language agents with tools in complex environments. Notably, equipped with the middleware, GPT-4 achieves **2.8**X the performance of the best baseline in tasks requiring access to database content and **2.2**X in KB tasks. Our findings illuminate the path for advancing language agents in real-world applications.

## introduction large language models (llms) have demonstrated a human-like mastery over text @xcite @xcite . however, the true ambition of ai extends well beyond the realm of text. the goal is to ultimately empower llms to act as generalist language agents that can aid humans across the multitude of complex realworld tasks @xcite @xcite , which often involve handling complex environments, be it browsing complex webpages @xcite , managing vast databases for llms to effectively serve as agents that ground human instructions accurately within the environment, they must develop a robust understanding of the environment. the most direct method to achieve it is to linearize the environment into a sequence of tokens that fit into the llm's short-term memory (i.e., its input window) and have the llm process the environment based on the linearized description @xcite @xcite . however, such a method faces steep challenges in scaling to more complex environments. also, discrete token descriptions may not reflect the most natural perception of the environment. recent work has explored using tools to extend the boundary of the llm's capacity @xcite @xcite . the core idea is that llms can actively decide a proper tool to use, using language as a powerful vehicle of thought @xcite . intuitively, we can also equip the llm with tools that enable navigating complex environments, so that the llm can proactively invoke different tools to explore the environment, thus circumventing limitations posed by its short-term memory (figure 1 ). however, this promising paradigm has been thus far underexplored. in this paper, we aim to delve into this paradigm and answer an intriguing question: how effectively can llms handle complex environments with the aid of tools? answering this question requires equipping the llm with a suite of tools designed to meet a wide range of needs within the target environment. in this paper, we carefully develop such tailored tools for two exemplar complex environments, i.e., databases and knowledge bases (kbs). unlike readily available web apis @xcite used in prior research, our tools have to be manually invented from scratch. in crafting these tools, we capitalize on the intuition of human informationgathering behaviors-such as performing keyword searches to identify a relevant database column or investigating the connections of a kb entity-to fulfill complex tasks in these environments (section 3.1). ideally, these tools are designed to function as a middleware layer between the llm and the environment, shielding the llm from environmental complexity. with these specialized tools, we propose two novel schemes to enable the llm to more accurately orchestrate its internal reasoning and tool usage: error feedback and decoupled generation (section 3.2). the combination of the crafted tools and the tool-use schemes allows the llm to actively explore the environment and ground human instructions into accurate actions. we evaluate different llms on benchmarks featuring complex tasks over the target environments, including a newly curated benchmark for the kb. the results are revealing: llms equipped with customized tools demonstrate a significant enhancement in their ability to engage with complex environments, markedly surpassing the prior art. in particular, despite its simplicity, such a middleware layer allows gpt-4 (openai, 2023a) to achieve 2.8× the performance (i.e., 38.3% vs. 13.8%) of the best baseline in tasks requiring access to database content and 2.2× (i.e., 59.3% vs. 27.1%) in kb tasks. our findings underscore the integral role of tool augmentation in enabling llms to handle complex environments. our main contributions are as follows: a) we develop a new framework with customized tools for two complex environments, to investigate the role of tools in handling complex environments with llms; b) we evaluate six different llms on our carefully chosen benchmarks for a comprehensive analysis; c) our analysis highlights a critical takeaway: augmenting llms with tools is crucial for successfully tackling complex environments, opening new possibilities to progress llms as generalist language agents for practical applications. ## related work interface complex environments with llms. existing methods that feed the environment directly into the llm for grounding @xcite would fail in complex environments due to scalability issues. specifically, these methods process the environment by linearizing it into discrete tokens @xcite @xcite @xcite . however, linearizing expansive environments like databases with millions of entries @xcite or lengthy webpage html code @xcite can often exceed an llm's input length constraints. alternative studies bypass the llm's direct interaction with complex environments by generating ungrounded draft plans for post-processing grounding @xcite or by using the llm to assess grounded plans created via predefined rules @xcite . such strategies do not fully utilize the llms' innate reasoning potential in actively navigating complex environments. in this paper, we explore a new paradigm where we can bypass these issues by equipping llms with a suite of comprehensive tools to actively gather necessary information about the environment upon demand, leveraging the llms' inherent reasoning capabilities. tool learning. tools are essential for enhancing the capabilities of llms @xcite @xcite . existing research, such as toolllm @xcite and api-bank @xcite , focuses on open-domain applications with a wide array of readily available restful apis. in contrast, this paper specifically aims to study the potential of tools in augmenting llms to effectively execute tasks within complex environments, where we carefully craft the specialized tools for different environments by ourselves. in addition, research focusing on restful apis typically displays shallow reasoning, while practical tasks within a complex environment typically entail a long sequence of actions (e.g., querying a kb or browsing a webpage). to enable tool use in more intricate settings within a more specific complex environment, structgpt @xcite employs a predefined sequence of tool invocations; chameleon @xcite functions in an open-loop setting where the llm directly produces a sequence for tool usage before any execution occurs. both of them fail to seamlessly integrate the reasoning capacity of the llm with the use of tools. in this paper, we propose two novel schemes-error feedback and decoupled generation to more seamlessly and accurately orchestrate the llm's internal reasoning and tool usage. ## middleware for llms we equip llms with a suite of tools specifically tailored to support an extensive variety of operations and cater to the diverse needs within a complex environment e. we call these tools middleware, as they can serve as a feature-rich middle layer between the llm and e, abstracting the llm from having to directly interact with all of its intricacies (section 3.1). furthermore, to fully unleash the inherent reasoning capabilities of llms in invoking proper tools, we propose two novel schemes to enhance tool use accuracy: error feedback, which provides concrete tool use error information and expects the llm to correct the error autonomously, and decoupled generation, where the llm's reasoning steps and tool use are separated for better controllability (section 3.2). this unified framework allows us to reliably investigate the potential of llms in handling complex environments with the aid of tools. ## benchmarks the predominant tasks for databases and kbs are text-to-sql parsing and kbqa. however, popular benchmarks for them may fall short for evaluating language agents out-of-box. specifically, the majority of questions in popular kbqa datasets like webqsp @xcite are one-hop or two-hop questions, for which we can effectively handle with existing semantic parsing methods @xcite . additionally, the databases featured in spider @xcite and wikisql @xcite have limited complexity in terms of both schema design and the num-ber of rows in the tables. this over-simplification enables the direct feeding of the database schema to the llm, achieving strong performance without the need to access the actual content of the database @xcite . therefore, we need different benchmarks with complex environments and instructions that better mirror the realworld situations language agents must handle (see statistics of our benchmarks in appendix b). databases for databases, we leverage bird @xcite , which is a recent dataset notable for its complexity, featuring intricate instructions over highly complex databases. there are originally two different settings in bird: with and without oracle knowledge, where the oracle knowledge supplies specific information about the target database needed to fulfill each task. for instance, "exclusively virtual refers to @xmath0. with such oracle knowledge, the complexity of the environments is substantially mitigated; it offers a shortcut for the task and eliminates the necessity for deep engagement with the database. this cheating setting is also unrealistic for practical applications. as a result, we stick to the setting without oracle knowledge. for each of the 1534 questions in bird's dev set, we manually label whether accessing the database content is necessary to compile the sql queries, noting that access is unnecessary if all mentioned values in a question exactly match database cells. this facilitates decomposing the language agent's performance based on questions that require deeper database engagement (496 questions) versus not (1038 questions) and enables finegrained insights into the llm's performance. in addition to execution accuracy (ex) used in bird, which determines if the execution results of the predicted sql match those of the ground truth sql, we also evaluate whether the predicted sql is a valid sql query (va). kbs we curate kbqa-agent, a new test set sourcing from existing kbqa datasets that contain complex questions. in particular, we selected 500 diverse questions that involve at least three relations, or two relations coupled with an aggregation function (i.e., counting or superlative). for each question, we annotate it with a ground truth sequence of actions based on the toolset defined by us. 3 specifically, kbqa-agent comprises questions from three kbqa datasets on freebase: grailqa @xcite , complexwebq @xcite , and graphq @xcite , ensuring a wide range of question types and sources. kbqa-agent is designed to be more representative of challenging, real-world scenarios compared to existing benchmarks (appendix b). it offers an ideal testbed for evaluating language agents in interacting with massive kbs. we assess this through two metrics: f1 of answer entities and validity (va), a binary met-ric evaluating the llm's ability to find an answer, whether correct or not. for the underlying llms, we primarily compare middleware with baseline methods using two of the most advanced llms-gpt-3.5-turbo-0613 (openai, 2023b) and gpt-4-0613 (openai, 2023a)-since our goal is investigating the full potential of tool-enhanced llms operating within complex environments. in addition, we also explore four open-source llms to more comprehensively evaluate our framework: llama2-7b-chat, llama2-13b-chat @xcite , mistral-7b-instruct-v0.2 @xcite , and mixtral 8×7b-instruct-v0.1 @xcite . baselines to fully understand the potential of tool augmentation for assisting llms in handling complex environments, we compare middleware against an array of strong baselines. for text-to-sql parsing, llms demonstrate a strong ability to compose sql queries when properly prompted with the database schema (i.e., api docs prompting @xcite ). this also represents the current state-of-the-art prompting-based method when oracle knowledge is not available on bird's leaderboard. in adddition, we also compare with more baselines on bird's leaderboard that originally did not submit their results using no oracle knowledge (see appendix d.1). for all methods on text-to-sql parsing, we adopt the zeroshot setting. unlike text-to-sql parsing, directly prompting llms does not generate reasonable outputs for kbqa due to the massive size of the kb schema. instead, existing kbqa methods based on llms typically follow two paradigms: either first generating an ungrounded program and then grounding the program to the kb schema afterwards @xcite , or gradually constructing a complex program and grounding it step by step @xcite . we compare middleware with the most representative work from each paradigm, namely kb-binder @xcite and pangu @xcite . we also include structgpt as an additional baseline for tool use. for all kbqa methods except kb-binder, we provide a one-shot demo to obtain more meaningful results. ## conclusion a pioneering vision is for language agents to assist humans in tackling intricate real-world tasks. this paper demonstrates that with meticulously-crafted tools acting as middleware between llms and complex environments, llms can substantially exceed current solutions. our results spotlight these specialized tools' indispensable role in unlocking the potential of llms within complex real-world tasks previously posing immense challenges.