A pp W orld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents (2024)

Abstract: Autonomous agents that address day-to-day digital tasks (e.g., ordering groceries for a household), must not only operate multiple apps (e.g., notes, messaging, shopping app) via APIs, but also generate rich code with complex control flow in an iterative manner based on their interaction with the environment. However, existing benchmarks for tool use are inadequate, as they only cover tasks that require a simple sequence of API calls. To remedy this gap, we built AppWorld Engine, a high-quality execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs and populated with realistic digital activities simulating the lives of ~100 fictitious users. We then created AppWorld Benchmark (40K lines of code), a suite of 750 natural, diverse, and challenging autonomous agent tasks requiring rich and interactive code generation. It supports robust programmatic evaluation with state-based unit tests, allowing for different ways of completing a task while also checking for unexpected changes, i.e., collateral damage. The state-of-the-art LLM, GPT4O, solves only ~49% of our ‘normal’ tasks and ~30% of ‘challenge’ tasks, while other models solve at least 16% fewer. This highlights the benchmark’s difficulty and AppWorld’s potential to push the frontiers of interactive coding agents.

## introduction the prospects and scope for automation in our digital lives are growing along with the continuing advances in instruction following, coding, and tooluse abilities of large language models (llms) @xcite @xcite . most day-to-day tasks in our digital lives are a series of complex activities over multiple applications interleaved with reasoning and decision-making over intermediate results. for example, even the mundane task of ordering groceries for a shared household turns into a complex task over multiple apps -finding a grocery list from a notepad app, checking roommates' requests on a messaging app, and ordering on a grocery app. automating this not only requires operating multiple apps via apis but also performing interactive reasoning and sequen-tial decision-making, while handling unexpected hurdles, and avoiding undesirable outcomes. consider the example in fig. 1 , where the agent is to launch a playlist with enough songs covering the user's workout duration for the current day. to do this, the agent first needs to write code (1st code block) to find and "read" (print) the note containing the workout schedule. only after having seen how the note is structured-seeing that duration is listed day-wise-can the agent write the necessary code (2nd code block), which involves finding the current (today's) day-of-the-week and extracting the associated duration. 3 to select a playlist, it must write rich code with for-loops and other control flow to iterate over playlists, aggregate duration of individual songs within playlists, and play one covering the workout duration (3rd code block). responsible development of such autonomous agents requires rigorous, reproducible, and robust evaluation using realistic tasks accounting for complexities and dynamics of real digital environments. however, existing benchmarks for tool-based solutions @xcite @xcite @xcite ) fall short in achieving this. they involve a linear sequence of 1-4 api calls, without the need for rich code or writing code based on interaction with the environment. further, they evaluate via comparison to a reference solution (using an llm or a human), which does not work well for complex tasks that admit many varying solutions. to fill this gap, we propose a new framework, dubbed appworld foot_3 (illustrated in fig. 2 ) consisting of two parts. (i) appworld engine provides a high-quality simulator of real-world apps and people using these apps, controlled via apis ( §2). (ii) appworld benchmark provides complex tasks and their evaluation suites, built using appworld ( §3). we make the following contributions: ## appworld engine our goal is to develop a realistic ecosystem of dayto-day apps with appropriate data, along with a fully controllable execution environment where agents can use apis to operate the apps autonomously with no real-world consequences. using software engineering best practices (e.g., following rest api design principles, rigorously unit testing apis, and documenting them in detail), we have built appworld engine, a high-quality, easy-to-use simulator of a digital world of apps and people. here, we describe its main components. ## appworld benchmark we target a collection of natural, diverse, and challenging tasks covering many day-to-day scenarios. to this end, we create task scenarios that serve as blueprints for instantiating multiple tasks. e.g., "i like the last {last-color} {apparel} i bought on amazon, repurchase the same in that size. prefer {preferred-color}, if available; otherwise, go with the same color.". we instantiate multiple tasks per scenario by carefully varying the placeholder values (apparel, last-color, and preferred-color) and starting states for each task. for each scenario, we build a task generator module (illustrative example in fig. 3 ; code example in app. j) that creates such tasks. each generator includes setup, evaluation, and solution programs as described below. ## benchmarking experiments we benchmark state-of-the-art open and closed llms on appworld using prompting methods designed for coding tasks and general decisionmaking. these are all iterative approaches that terminate on calling a special task_completed api or when they hit their termination threshold 21 . more method details are in app. e. ## related work appworld introduces complex tasks requiring agents to combine tool usage and interactive code generation for app-based tasks. 26 tool-usage benchmarks: while some benchmarks do not equip agents with executable tools @xcite @xcite , the others do so either by utilizing existing public apis @xcite @xcite or providing implementations of simple ones @xcite @xcite . in comparison, appworld offers executable apis and tasks of much higher complexity and provides much more reliable and programmatic evaluation. while prior works require only 1-4 api calls to solve a task, appworld requires up to 26 apis (avg. 9.5) with dependencies between calls (i.e., outputs of calls used as inputs to subsequent calls), embedding calls into various programming structures with up to 134 (avg. 41. ## conclusions responsible development of llm-based autonomous agents for day-to-day tasks has been hindered by a lack of a unified, stable, and standardized executable environment and an absence of benchmarks with appropriate levels of complexity and interactivity. to fill this gap, we introduced the appworld framework with its controllable and reproducible execution environment and a benchmark of interactive coding for api-based complex everyday tasks. our programmatic evaluation suite and realistic hurdles and distractors allow a rigorous and robust assessment of autonomous agents. our benchmarking and analysis of state-ofthe-art models underscore the overall difficulty of appworld and highlight open challenges for modern llms to automate our everyday digital tasks. more broadly, appworld engine is a modular and extensible foundation that opens up many exciting possibilities in automating digital tasks, including extensions for ui-based control, supporting tasks requiring multi-agent (and human) coordination and collaboration, and serving as a sandbox to study potential privacy and safety risks that may arise when giving digital assistants the "agency" to act on our behalf in the real world.