T ime A rena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulation (2024)

Abstract: Despite remarkable advancements in emulating human-like behavior through Large Language Models (LLMs), current textual simulations do not adequately address the notion of time. To this end, we introduce TimeArena, a novel textual simulated environment that incorporates complex temporal dynamics and constraints that better reflect real-life planning scenarios. In TimeArena, agents are asked to complete multiple tasks as soon as possible, allowing for parallel processing to save time. We implement the dependency between actions, the time duration for each action, and the occupancy of the agent and the objects in the environment. TimeArena grounds to 30 real-world tasks in cooking, household activity, and laboratory work. We conduct extensive experiments with various LLMs using TimeArena. Our findings reveal that even the most powerful models, e.g., GPT-4, still lag behind humans in effective multitasking, underscoring the need for enhanced temporal awareness in the development of language agents.

## introduction large language models (llms) @xcite @xcite have enabled the development of language agents (a.k.a. llm-based agents), which aim to simulate human behaviors in real-world scenarios through their planning capabilities @xcite @xcite . however, planning in the real world involves temporal and resource constraints @xcite , which are rarely implemented in most textual simulations for llms and language agents @xcite . the integration of time in simulated environments challenges agents to navigate and align with human-like efficient multitasking skills. such a simulation requires the agent to consider the following three factors: 1) time duration and dependency: actions will have durations upon dependencies, requiring agents to strategize and prioritize based on time constraints and task completion progress. 2) agent occupancy: agents will be occupied by certain actions; thus, they might be unable to perform other actions at the same time. in this work, we introduce timearena, a textual simulated environment featuring 30 real-world tasks involving cooking, household activity, and laboratory work. timearena is the first textual simulation to evaluate language agents on multitasking efficiency. specifically, we incorporate the time duration of each action and set two types of actions based on agent occupancy. one type occupies agents (e.g., wash cup) and another lets agents be idle (e.g., boil water). additionally, we simulate resource competition by implementing object occupancy, i.e., an object used for one task cannot be simultaneously used for another, which is common in parallel processing. therefore, agents must focus on parallel processing, taking into account the occupancy of agents and objects, to minimize time consumption. we design four metrics in timearena to evaluate the average progress score, completion speed, task completion rate and average completion time. these metrics help to assess and analyze the efficient multitasking capabilities of language agents. our comprehensive evaluation of seven llms on timearena shows that current language agents struggle in efficient multitasking. even the most powerful llm, gpt-4, still faces challenges in parallel processing. in summary, our contributions are as follows: ## related work simulation-based evaluation for language agents with the great success of llms (ope-nai, 2022, 2023; team and google, 2023), recent works have shifted the focus from traditional nlp tasks to explore language agents in simulated environments that mimic real-world scenarios @xcite @xcite . these simulated environments can be divided into two categories: 1) social simulations @xcite @xcite , which aim to evaluate the behaviors of language agents in some social scenarios; 2) problem-solving simulations, which are created based on games @xcite @xcite and scientific scenarios @xcite . in this paper, we focus on problem-solving simulations to investigate the efficient multitasking capabilities of language agents. language planning language planning aims to decompose a complex task into steps @xcite . early studies mainly focus on imbuing language models with planning capabilities by training them on specific planning datasets @xcite @xcite , which exhibits poor generalization. recent studies have identified that llms can effectively decompose tasks into procedural steps @xcite @xcite . however, multitasking planning with parallel processing in dynamic environments still remains under-studied. temporal reasoning temporal reasoning involves comprehending, structuring, and interpreting events, actions, and states through the lens of time @xcite @xcite . previous studies in temporal reasoning focus on temporal relation extraction @xcite @xcite , event temporal reasoning @xcite @xcite and explore the temporal reasoning capability of llms with several contemporary time-sensitive qa datasets (zhang simulation designs objective: completing 3 tasks in the shortest time timeline: agent interacting with the environment wash clothes in washing machine (20min) hang clothes (5min) wash cup (5min) wash teapot (4min) boil water (8min) make tea (3min) pour tea into cup (2min) wash bed sheet in washing machine (15min) hang bed sheet (3min) @xmath0, boil water does not agent action environment feedback task1: make tea task2: wash clothes task3: wash bed sheet ⏰ boiling water washing clothes ⏰ ⏰ washing bed sheet ⏰ ⏰ making tea ⏰ ⏰ making tea ⏰ ⏰ waiting ⏰ ⏰ washing teapot ⏰ ⏰ ⏰ … washing teapot ⏰ ⏰ ⏰ wash teapot (4min) boil water finished. wash teapot finished. make tea (3 min) wait (1 min) wash clothes (20 min) boil water (8 min) … task dependency agent occupancy object occupancy washing machine is occupied! teapot is dirty and water has not boiled! ## timearena we create timearena, a textual simulated environment to evaluate the efficient multitasking capabilities of language agents. to help illustrate, we first show an overview and an example run of how an agent interacts with the timearena environment ( § 3.1), and then describe the design of the simulation environment in more detail ( § 3.2 and § 3.3). ## conclusion in this paper, we introduce timearena, a textbased simulated environment designed to incorporate the notion of time. timearena extends beyond simply acknowledging the dependency of actions by also considering their duration, an essential factor in time modeling. using timearena, we evaluate the multitasking and parallel processing capability of language agents. our findings indicate that language agents still have significant room for improvement when completing multiple tasks in dynamic environments in a minimal time, highlighting an area for future research.