One Agent To Rule Them All: Towards Multi-agent Conversational AI (2022)

Abstract: The increasing volume of commercially available conversational agents (CAs) on the market has resulted in users being burdened with learning and adopting multiple agents to accomplish their tasks. Though prior work has explored supporting a multitude of domains within the design of a single agent, the interaction experience suffers due to the large action space of desired capabilities. To address these problems, we introduce a new task BBAI: Black-Box Agent Integration, focusing on combining the capabilities of multiple black-box CAs at scale. We explore two techniques: question agent pairing and question response pairing aimed at resolving this task. Leveraging these techniques, we design One For All (OFA), a scalable system that provides a unified interface to interact with multiple CAs. Additionally, we introduce MARS: Multi-Agent Response Selection, a new encoder model for question response pairing that jointly encodes user question and agent response pairs. We demonstrate that OFA is able to automatically and accurately integrate an ensemble of commercially available CAs spanning disparate domains. Specifically, using the MARS encoder we achieve the highest accuracy on our BBAI task, outperforming strong baselines.

## introduction influenced by the popularity of intelligent conversational agents (cas), such as apple siri and amazon alexa, the conversational ai market is growing at an increasingly rapid pace and is projected to reach a valuation of us @xmath00.12 for 5 utterances. these submitted utterances were then vetted by hand to ensure quality. using the curated utterances, we then generated question responses by querying each agent to gather its response to the utterance. in order to generate ground truth samples on which of the question-response pairs (q, r i ) correctly resolves the query q we launched a crowdsourcing task asking workers to indicate the candidate responses that best answer the question shown. five workers were assigned to each response selection task and majority voting (>2) was used to label the gold responses. as such for each query q and the set of responses r we were able to gather the necessary question-agent pairs (q, a i ) and question-response pairs (q, r i ) needed to evaluate our approaches. ## results and discussion in this section we present and analyze the results of our experiments, detailing our insights and discussing the implications of each of our techniques. evaluation task: similar to standard information retrieval evaluation measures, we denote accuracy as the metric precision@1 and use it to evaluate both our question agent and question response pairing approaches. for question agent pairing this metric denotes: given a set of n agents to the given query, whether the agent selected ultimately resolves the query successfully. for question response pairing it denotes: given a set of n responses to the given query, whether the top-scoring response resolves the query successfully. for this evaluation, we test on examples with at least one valid agent response. ## related work ensemble approaches to solving complex tasks in the context of nlp are widely used @xcite . in dialogue systems, recent attempts at ensemble approaches and multi-agent architectures include cercas @xcite and @xcite . alanav2 (cercas @xcite demonstrated an ensemble architecture of multiple bots using a combination of rule-based machine learning systems built to support topic-based conversations across domains. it was built to be an open domain bot supporting topic-based conversations. specifically, alanav2's architecture utilizes a variety of ontologies and nlu pipelines that draw information from a variety of web sources such as reddit. however, its agent selection approach is guided by a simple priority bot list. @xcite describe their conversational framework that employs an orchestrator bot to understand the user query and direct them to a domain-specific bot that handles subsequent dialogue. in our work, we expand up the multi-agent goal by focusing on the integration of black-box conversational agents at scale. ## conclusion the rapid proliferation of conversational agents calls for a unified approach to interacting with multiple cas. the key challenge of building such an interface lie in that most commercial cas are black-boxes with hidden internals. this paper introduces bbai a new task of agent integration that focuses on unifying black-boxes cas across varying domains. we explore two task techniques, question agent pairing and question response pairing and present one for all, a scalable system that unifies multiple black-box cas with a centralized user interface. using a combination of commercially available conversational agents, we evaluate a variety of approaches to multi-agent integration through one for all. our mars encoder achieves 83.5% accuracy on bbai and outperforms the best single agent configuration by over 32%. these results demonstrate the power of one for all which can leverage state-of-the-art nlu approaches to enable multiple agents to collectively achieve more than any single conversational agent in isolation eliminating the need for users to learn and adopt multiple agents. this work opens up a wide range of potential future work involving the design of systems geared towards facilitating more advanced multi-agent interaction. we foresee a system with even greater response selection performance as the nlp community continues to produce more state-of-the-art language models with even greater contextual knowledge of the world. extensions of this work can include examining not only the integration of agents but the interoperability by facilitating the passing of shared conversation knowledge across agents especially in multi-turn conversational scenarios across multiple agents.