M eta QA : Combining Expert Agents for Multi-Skill Question Answering (2023)

Abstract: The recent explosion of question-answering (QA) datasets and models has increased the interest in the generalization of models across multiple domains and formats by either training on multiple datasets or combining multiple models. Despite the promising results of multi-dataset models, some domains or QA formats may require specific architectures, and thus the adaptability of these models might be limited. In addition, current approaches for combining models disregard cues such as question-answer compatibility. In this work, we propose to combine expert agents with a novel, flexible, and training-efficient architecture that considers questions, answer predictions, and answer-prediction confidence scores to select the best answer among a list of answer predictions. Through quantitative and qualitative experiments, we show that our model i) creates a collaboration between agents that outperforms previous multi-agent and multi-dataset approaches, ii) is highly data-efficient to train, and iii) can be adapted to any QA format. We release our code and a dataset of answer predictions from expert agents for 16 QA datasets to foster future research of multi-agent systems.

## introduction the large number of question answering (qa) datasets released in the past years has been accompanied by models specialized in them @xcite . these datasets and models differ by the domain (e.g., biomedical and wikipedia), required skills (e.g., numerical and multi-hop), and format (e.g., extractive and multiple-choice). this variety of tasks and overspecialization of the corresponding models have led the community towards developing simple unified models that can generalize across domains and formats through unifying dataset formats @xcite , creating models trained on multiple datasets @xcite @xcite , and designing ensemble methods for qa agents @xcite . all these research lines have a potential impact on end-user applications because generalization can help create robust systems and ease the implementation of qa models. for example, some chatbots are composed of skill systems, where each skill is a model trained on a specific domain @xcite . more abstractly, these research lines also share a central research question: how to combine qa skills. we argue that a one-size-fits-all architecture may encounter some limitations in combining qa skills. for instance, @xcite have observed that a single model trained on multiple tasks may underperform the same architecture trained on a single task. an alternative approach is to combine multiple expert agents. @xcite propose a model that given a question and a list of datasets, selects the dataset from which the question comes. this can be used to identify agents trained on a specific type of questions. however, despite achieving a classification accuracy greater than 90%, this approach underestimates high-performing models on out-of-domain questions. to address the limitations of previous approaches, we propose a novel model, metaqa, to combine heterogeneous expert agents (i.e., different architectures, formats, and tasks). it takes a question, and a list of candidate answers with confidence scores as input and selects the best answer (figure 1 ). we modify the embedding mechanism of the transformer encoder @xcite to embed the confidence score of each candidate answer. in addition, we use a multi-task training objective that makes the model learn two complementary tasks: selecting the best candidate answer and identifying agents trained on the domain of the input question. our approach learns to match questions with answers, an immensely easier task than the endto-end qa of multi-dataset models. this makes metaqa remarkably data efficient as it only uses 16% of the training data of multi-dataset models. we compile a list of 16 qa datasets that encompass different domains, formats, and reasoning skills to conduct experiments. through quantitative experiments, we show that our metaqa i) establishes a successful collaboration between agents, ii) outperforms multi-agent and multi-dataset models, iii) excels in minority domains, and iv) is highly efficient to train. our contributions are: currently, there are two approaches for multi-skill qa: multi-agent and multi-dataset models. multi-agent models consists of combining multiple expert agents. a well-known method is the mixture of experts. it requires training a set of models and combining their outputs with a gating mechanism @xcite . however, this approach would require jointly training multiple agents, which can be extremely expensive, and sharing a common output space to combine the agents. these limitations make it unfeasible to implement in our setup, where many heterogeneous agents are combined (i.e., agents with different architectures, target tasks, and output formats such as integers for multiple-choice or answer spans for span extraction). inspired by topic classification, @xcite proposed mapping questions to qa datasets (topics) to identify agents trained on that type of questions. although related to us, their work does not attempt to achieve any agent collaboration. moreover, because of their topic-classification approach, agents that are effective in out-of-domain questions are underestimated. lastly, @xcite average the weights of adapters @xcite trained on single datasets to obtain a multi-dataset model. however, their architecture is limited to span extraction. multi-dataset models consist of training a model on various datasets to generalize it to multiple domains. @xcite conduct extensive analyses of the generalization of qa models. however, they only experiment on extractive tasks and, due to their model architecture (bert for span extraction), it is not possible to extend it to other tasks such as abstractive or visual qa. @xcite created a competition on qa generalization using 18 datasets. these datasets are from very different domains, such as wikipedia and biomedicine, among others. however, they also focus only on extractive datasets. lastly, @xcite shows that the different qa formats can complement each other to achieve a better generalization. they use an encoder-decoder architecture and transform the questions into a common format. however, we argue that their approach is limited because some questions may require a specific skill that must be modeled in a particular manner (e.g., numerical reasoning), and this is not possible with their simple encoder-decoder. ## model we propose a new model, shown in figure 2 , to combine k qa agents m. each agent m i is trained on domain dom i and predicts an answer ans i . without loss of generalizability, we assume that each agent is trained on a different domain and each question belongs to one of these domains. we define two complementary tasks: i) a priori agent selection (agent selection networks, agsen, in figure 2 ) and ii) answer selection (anssel network in figure 2 ). the division of the problem into these two learnable tasks is vital to ensure that metaqa considers out-of-domain agents, which can also give correct answers. more formally, metaqa computes the probability of returning an answer a as follows: to achieve this, the backbone of our architecture relies on an encoder transformer @xcite whose input is the concatenation of the question with the candidate answers from each agent. each answer is separated by a new token [ans] that informs the model of the beginning of a new answer candidate. we devise a new embedding for the transformer encoder to include the confidence score of the pre-dictions of each agent (figure 3 ). while the original encoder uses the token t i , position p i , and segment s i embeddings, we add an agent confidence embedding c i to these three. the new c i is obtained with a feed-forward network f that takes an answer confidence conf i and creates an embedding c i . ## results and discussions in this section, we answer the questions: i) is metaqa able to combine multiple agents without undermining the performance of each one ( §5.1), ii) is it robust on out-of-domain scenarios? ( §5.2), iii) how does agent collaboration work? ( §5.3), iv) how data-efficient is metaqa? ( §5.4), and v) what is the effect of each module of metaqa? ( §5.5). ## conclusions in this work, we propose an alternative to multidataset models for multi-skill qa. we propose to combine expert agents to create a collaborative system for question answering (qa) called metaqa. it considers questions, answer predictions, and confidence scores from the agents to select the best answer to a question. through quantitative experiments, we show that our model avoids the limitations of multi-dataset models and outperforms the baselines thanks to the agent collaboration established. additionally, since metaqa learns to match questions with answers instead of end-to-end qa, it is highly data-efficient to train. we leave as future work: i) combining partially correct answer predictions to generate a better one, ii) adding new agents without retraining metaqa by fixing most of the weights and only training the weights of the new agent selection network, and iii) identifying a priori agents that are likely to give an incorrect answer to skip them at run-time. in appendix a.1. we are not held responsible for errors, false or offensive content generated by the agents. metaqa should be used at the users' discretion. future work should address how to identify unfair or false content to avoid selecting it.