Hey AI , Can You Solve Complex Tasks by Talking to Agents? (2022)

Abstract: Training giant models from scratch for each complex task is resource- and data-inefficient. To help develop models that can leverage existing systems, we propose a new challenge: Learning to solve complex tasks by communicating with existing agents (or models) in natural language. We design a synthetic benchmark, CommaQA, with three complex reasoning tasks (explicit, implicit, numeric) designed to be solved by communicating with existing QA agents. For instance, using text and table QA agents to answer questions such as “Who had the longest javelin throw from USA?”. We show that black-box models struggle to learn this task from scratch (accuracy under 50%) even with access to each agent’s knowledge and gold facts supervision. In contrast, models that learn to communicate with agents outperform black-box models, reaching scores of 100% when given gold decomposition supervision. However, we show that the challenge of learning to solve complex tasks by communicating with existing agents without relying on any auxiliary supervision or data still remains highly elusive. We will release CommaQA, along with a compositional generalization test split, to advance research in this direction.

## introduction a common research avenue pursued these days is to train monolithic language models with billions of parameters @xcite @xcite to solve every language understanding and reasoning challenge @xcite . in contrast, humans often tackle complex tasks by breaking them down into simpler subtasks, and solving these by interacting with other people or automated agents whose skill-sets we are familiar with. this approach allows us to learn to solve new complex tasks quickly and effectively, by building upon what's already known. can ai systems learn to do the same? to facilitate research in this direction, we propose a new reasoning challenge and a benchmark called commaqa where, in addition to the usual end-task supervision, one has access to a set of predefined ai agents with examples of their natural language inputs. importantly, the target end-task is designed to be too difficult for current models to learn based only on end-task supervision. the goal is instead to build models that learn to solve the target task by decomposing it into sub-tasks solvable by these agents, and interacting with these agents in natural language to do so. as a motivating example, consider the interaction depicted in figure 1 where a system is asked to buy a book series with a certain property. the system breaks this goal down, using agent-1 (here google assistant) to identify the referenced book series as well as the list of books in that series, and then using agent-2 (here amazon alexa) to make the purchase. while both of these agents interact with the system in natural language, they have different and complementary skill sets, 2 rely on privately held knowledge sources, and have been built at an enormous cost. at the same time, neither agent by itself can accomplish the original goal. an alternative to building such a system that interacts with existing agents is to teach all requisite sub-tasks and skills to a large black-box system, say via multi-task learning @xcite . this, however, not only wastes time and resources, but is often also infeasible. for example, agents such as google assistant and ope-nai gpt-3 use private knowledge resources and are computationally expensive to train even once. it would thus be nearly impossible to build a single system with the capabilities of both of these agents. we note that agents need not be sophisticated ai assistants. an agent may simply be a previously developed question-answering (qa) model, a math module, a function of textual input, an image captioning system-anything the community already knows how to build. the goal is to learn to leverage existing agents for more complex tasks. to enable the development of general systems for this task, we identify the minimal inputs that must be assumed for the task to be learnabletraining data for the complex task, existing agents that together can solve the complex task, and examples of valid questions that can be asked of these agents (capturing the agents' capabilities). we build a new synthetic benchmark dataset called commaqa (communicating with agents for qa), containing three complex multihop qa tasks (involving explicit, implicit, and numeric reasoning) and four input qa agents that can solve these tasks. commaqa is not yet another multi-hop reading comprehension dataset. it is designed to facilitate the development of a new family of techniques that teach systems to communicate with a wide variety of agents to solve different types of complex tasks. we demonstrate that black-box models struggle on commaqa even when provided with auxiliary data, such as domain-relevant agent knowledge. on the other hand, a model that leverages the agents @xcite can achieve very high accuracy but relies on auxiliary supervision (decomposition annotations). while it is possible to identify valid decompositions using just the endtask labels, the search space is extremely large and naïve approaches, as we show, help only with one of the datasets. commaqa thus serves as a new challenge for the nlp community. ## challenge task definition we formalize the new challenge task of learning to talk with agents to solve complex tasks. to ensure generality of solutions, we identify minimal inputs for the task to be well-defined and learnable. first we must define {f i } m @xmath0, the agents or models that solve simpler sub-tasks. 4 minimally, we need to define the space of valid inputs l i for each agent f i , i.e., how can they be invoked. for a system to identify the appropriate agent for each subtask, we also need to define the capabilities of each agent. since these agents are often defined for natural language tasks, the space of inputs captures the capabilities of these agents too. for instance, "buy the book 'harry potter and the sorcerer's stone'" captures the alexa agent's capability of buying books. instead of complex formal specifications of the agent's capabilities, we use natural language inputs as a rich and convenient representation. next, we need a target task t that can be solved via a composition of the capabilities of various f i . 5 finally, to pose this as a machine learning problem, we need training data @xmath1, y k )} n @xmath2. since collecting annotations for complex tasks can be difficult, d is expected to be relatively small. models must therefore use the available agents, instead of learning the complex task from scratch. given these pre-requisites, we can define the challenge task as follows: challenge: learn a model to solve a complex task t , given only: -training dataset @xmath3, y k )} n @xmath4; -agents {f 1 , . . . , f m } that can help solve t ; -examples from the space l i of valid inputs for each agent f i that captures its capabilities. one example of this challenge is answering multi-hop questions given two agents: an opendomain textqa agent f 1 and an open-domain tableqa agent f 2 . agent f 1 can use large textual corpora to answer questions such as "who directed kill bill?". agent f 2 can use tables (e.g., filmography tables) to answer questions such as "list the movies directed by quentin tarantino". finally, the training data t for the complex task would contain examples such as ("what movies has the director of kill bill appeared in?", ["reservoir dogs", ...,]). ## dataset: commaqa benchmark we next propose a new benchmark dataset com-maqa that enables the development of models that can learn to communicate with existing agents. specifically, we provide a collection of three synthetic datasets where each question is answerable by talking to simple qa agents. note that we are not proposing a new class of questions but a new dataset for the proposed challenge task. a highlevel overview of this dataset is shown in fig. 2 . we choose qa as the underlying task and use qa agents for this challenge because the question-answer format can capture a broad range of tasks @xcite while also naturally surfacing the capability of each agent. for instance, the question "what are the key frames in v?" describes a capability of the invoked agent (namely, identifying key frames), in addition to the specific inputs. we next describe our framework for build-pludgel, dessication hey tableqa, which awards were given to pneumodendron? q: what awards have movies written by people born in 1905 won? a: ["pludgel", "dessication", "pianogram"] q: what awards have movies written by people born in 1905 won? a: ["pludgel", "dessication", "pianogram"] what movies has alpinista written? which awards were given to vitrilateral? which movies were released in 1957? ... who is from the country schelpla? which awards were given to fidelice? who were born in the year 1923? ... ## experiments we evaluate various models on commaqa, including a baseline model (with no auxiliary information) for the task and state-of-the-art models that have access to auxiliary information. ## closing remarks we motivated a new challenge of solving complex tasks by communicating with existing ai agents. this challenge, we believe, will help develop more generalizable and efficient models. we introduced a new benchmark dataset commaqa with three multi-hop reasoning challenges, all solvable by composing four qa agents. state-of-the-art language models struggle to solve commaqa, even when provided with agents' internal knowledge. in contrast, a model that is able to learn to communicate with the agents, albeit using annotated decompositions, is able to solve this task. these results point to the need for and the potential of such approaches, but without reliance on auxiliary annotations, to solve complex tasks. commaqa is only one instantiation of our overall framework. one can extend it in many ways, such as using lms to enrich lexical diversity, emulating the behavior of imperfect real-world agents that even attempt to answer out-of-scope questions, diversifying to other reasoning types such as boolean questions where using distant supervision is even harder @xcite , and extending the generalization dataset to include new examples of valid inputs as well as new agents.