MARCO : Multi-Agent Real-time Chat Orchestration (2024)

Abstract: Large language model advancements have enabled the development of multi-agent frameworks to tackle complex, real-world problems such as to automate workflows that require interactions with diverse tools, reasoning, and human collaboration. We present MARCO, a Multi-Agent Real-time Chat Orchestration framework for automating workflows using LLMs. MARCO addresses key challenges in utilizing LLMs for complex, multi-step task execution in a production environment. It incorporates robust guardrails to steer LLM behavior, validate outputs, and recover from errors that stem from inconsistent output formatting, function and parameter hallucination, and lack of domain knowledge. Through extensive experiments we demonstrate MARCO’s superior performance with 94.48% and 92.74% accuracy on task execution for Digital Restaurant Service Platform conversations and Retail conversations datasets respectively along with 44.91% improved latency and 33.71% cost reduction in a production setting. We also report effects of guardrails in performance gain along with comparisons of various LLM models, both open-source and proprietary. The modular and generic design of MARCO allows it to be adapted for automating workflows across domains and to execute complex tasks through multi-turn interactions.

## introduction advancements in llm technology has led to a lot of interest in applying agents framework to realise solutions which require complex interactions with the environment including planning, tools usage, reasoning, interaction with humans. recent works @xcite demonstrate potential of llms for creating autonomous agents while there are numerous challenges to overcome and provide a seamless experience for end users who interact with the system at a daily basis. llms are probabilistic next token prediction systems and by design, non-deterministic which can introduce inconsistencies in the output generation that can prove challenging for features like function calling, parameter value grounding, etc. there are also challenges of domain specific knowledge which can be an advantage and disadvantage at the same time. llms have biases inherent in them which can lead to hallucinations, at the same time it may not have the right internal domain specific context which needs to be provided to get the expected results from an llm. we present our work on building a real time conversational task automation assistant framework with the following emphasis, (1) multi-turn interface for, (a) user conversation to execute tasks (b) executing tools with deterministic graphs providing status updates, intermediate results and requests to fetch additional inputs or clarify from user. (2) controllable agents using a symbolic plan expressed in natural language task execution procedure (tep) to guide the agents through the conversation and steps required to solve the task (3) shared hybrid memory structure, with long term memory shared across agents which stores complete context information with agent teps, tool updates, dynamic information and conversation turns. (4) guardrails for ensuring correctness of tool invocations, recover for common llm error conditions using reflection and to ensure general safety of the system. (5) evaluation mechanism for different aspects and tasks of a multi-agent system. this is demonstrated in the context of task automation assistant which supports adding usecase tasks to provide users a conversational interface where they can perform their intended actions, making it easier for them to refer to informational documents, interact with multiple tools, perform actions on them while unifying their interfaces. we provide detailed comparison across multiple foundational llms as backbone for our assistant tasks like claude family models @xcite , mis-tral family models @xcite and llama-3-8b (ai@meta, 2024) on digital restaurant service platform (drsp) conversations and retail conversations (retail-conv) datasets. ## related work improvements to llm technology through the release of foundational llms like @xcite , claude @xcite and mixtral @xcite has led to a flurry of research around autonomous agents and frameworks @xcite . zero shot chainof-thought (cot) reasoning @xcite allows llms to perform task reasoning by making it think step by step. llms can invoke external tools based on natural language instructions. hugginggpt @xcite can coin series of model invocations to achieve complex tasks mentioned by the user. toolformer @xcite demonstrates how llms can be used as external tools through api invocations selecting the right arguments to be passed from few examples and textual instructions. agents framework @xcite discuss using natural language symbolic plans called (standard operating procedures) sops which define transition rules between states as the agent encounters different situations to provide more control over agent behavior along with memory to store relevant state information within the prompt @xcite or long term context externally @xcite . amazon bedrock agents 1 provide interface to quickly build, configure and deploy autonomous agents into business applications leveraging the strength of foundational models, while the framework abstracts the agent prompt, memory, security and api invocations. langgraph 2 is an extension of langchain which facilitates the creation of stateful, multi-actor applications using large language models (llms) by adding cycles and persistence to llm applications thus enhancing their agentic behavior. it coordinates and checkpoints multiple chains (or actors) across cyclic computational steps. while these frameworks present novel ways for llms to act in a desired behaviour, they often have accuracy-latency trade-off where to improve on the accuracy the system latency increases due to multi-step planning and thinking @xcite . our proposed solution, 1 amazon bedrock agents user guide 2 langgraph library marco, not only interacts with user in a multiturn fashion but also has multi-turn conversation with deterministic multi-step functions which comprises of pre-determined business logic or task execution procedure (tep) requiring agents only at intelligent intervention related steps. along with the usecase teps, multi-step functions and robust guardrails to steer llm behaviour, marco is able to perform complex tasks with high accuracy in less time as detailed in subsequent sections. ## marco: multi-agent real-time chat orchestration in this section, we discuss our approach for marco. section 3.1 formulates the problem statement in terms of task automation via real-time chat, followed by components of marco in section 3.2 and the evaluation methods on performance and latency for marco in section 3.3. ## experimental setup dataset: for our experiments, we curated two conversational orchestration test datasets, digital restaurant service platform (drsp-conv) and retail-conv, each with 221 and 350 multi-turn conversations in the restaurant services and retail services domain respectively. these conversations are a mix of out-of-domain (ood), action and info queries with multi-turn interactions with both, user and deterministic tasks (an example conversation flow in the dataset is shown in figure 1 for drsp-conv). the dataset covers usecases along with their natural language task execution procedure (tep) steps, supported functions (deterministic tasks and utility tools 4 ) and sub-task agents. each test conversation has multiple assistant (agent) messages (replying to the user, loading an agent, calling a deterministic task, or answering an informational query). we use these datasets for evaluating marco on the defined performance metrics. hyper-parameter details mentioned in appendix a.2. baseline: we implement marco with a single agent-based prompt as a baseline to compare with our multi-agent proposed solution, on performance, latency and cost. to achieve this, the usecase subtask tep steps in the datasets were combined into the parent agent tep steps to create a single agent tep and the datasets were modified accordingly to support the single agent baseline. ## experiments & results effects of reflection guardrails: through this experiment, we compare mars's performance when all reflection guardrails, as discussed in section 3.2.3, are added vs without adding any guardrails. as shown in