UKP - SQ u ARE v3: A Platform for Multi-Agent QA Research (2023)

Abstract: The continuous development of Question Answering (QA) datasets has drawn the research community’s attention toward multi-domain models. A popular approach is to use multi-dataset models, which are models trained on multiple datasets to learn their regularities and prevent overfitting to a single dataset. However, with the proliferation of QA models in online repositories such as GitHub or Hugging Face, an alternative is becoming viable. Recent works have demonstrated that combining expert agents can yield large performance gains over multi-dataset models. To ease research in multi-agent models, we extend UKP-SQuARE, an online platform for QA research, to support three families of multi-agent systems: i) agent selection, ii) early-fusion of agents, and iii) late-fusion of agents. We conduct experiments to evaluate their inference speed and discuss the performance vs. speed trade-off compared to multi-dataset models. UKP-SQuARE is open-source and publicly available.

## introduction the current high-speed development of artificial intelligence yields thousands of datasets and trained models in repositories such as github and hugging face @xcite . these models are creating new research and application opportunities, such as high-performing question answering (qa) skills in chatbots @xcite . comparing and analyzing these models usually requires learning libraries, writing code to run the models, and unifying their formats to compare them, which makes this process timeconsuming and not scalable. ukp-square @xcite addresses this challenge, providing the first online platform that offers an ecosys-1 https://github.com/ukp-square/square-core tem for qa research enabling reproducibility, analysis, and comparison of qa models through a standardized interface and from multiple angles (i.e., general behavior, explainability, adversarial attacks, and behavioral tests). the large variety of tasks and domains in qa datasets is pushing the research community towards creating models that generalize across domains @xcite @xcite . currently, there are two main approaches to achieve this: i) multi-dataset models and ii) multi-agent models. while the former trains a model on multiple datasets @xcite , the latter combines multiple expert agents @xcite @xcite . concurrently, large language models (llm) such as @xcite are emerging as new powerful systems for multi-task and multi-domain nlp applications. these llm models are complementary to the focus of our work, multi-agent systems. while llms show impressive performance, they are extremely expensive to run and can usually only be accessed through apis or deployed with great hardware resources. on the other hand, multi-agent systems offer a solution to create multi-domain models reusing available pretrained models that can be run on more modest hardware, which is an important requirement, e.g. where data cannot be sent to third parties. multi-agent models are particularly promising due to the thousands of models readily available on online model hubs and their current exponential growth. 2 this growth in the number of models is increasing the interest of the community in multiagent model research @xcite @xcite @xcite @xcite . however, model hubs such as hugging face only allow inference on individ- metaqa made question tweac squad skill news skill trivia skill avg. weights squad adapter news adapter trivia adapter answer question answer squad skill news skill trivia skill answer answer answer question answer dataset score skill selection (tweac) early-fusion (made) late-fusion (metaqa) figure 1: overview of different multi-agent system architectures deployed in ukp-square. tweac (left) selects an agent (a skill in ukp-square) based on which dataset it predicts the question is closest to and on which dataset a skill was trained. made (center) fuses the weights of adapters trained on different datasets. metaqa (right) ## ukp-square ukp-square @xcite is the first online platform that offers an ecosystem for qa research. its goal is to provide a common place to share, run, compare, and analyze qa models from multiple angles, such as explainability, adversarial attacks, behavioral tests, and i/o behaviors. the platform follows a flexible and scalable microservice architecture containing five main services: all these services allow ukp-square to offer an ecosystem of tools to analyze skills through a user-friendly interface without writing any code or complex configurations. ukp-square helps researchers identify the models' strengths and weaknesses to push the boundaries of qa research. ## multi-agent systems multi-agent systems are a type of multi-domain system that aggregate multiple expert agents from different domains to create a unified system. i.e., their focus is on the agents (skills in ukp-square). on the other hand, multi-dataset systems aim to learn a unified model from multiple data distributions to create a single, general agent. for example, unifiedqa @xcite is a qa model trained on multiple datasets using a generative model to overcome format boundaries. however, @xcite show that a model trained on multiple datasets may underperform the same architecture trained on a single dataset, i.e., multi-dataset models may underfit certain distributions. based on this observation, @xcite show that multi-agent models can avoid this limitation while being data-efficient to train and even outperform multi-dataset models by large margins in both in-domain and out-of-domain scenarios. this is possible because instead of using a very general architecture to solve multiple tasks, it uses a list of expert agents with specific architectures designed to solve those tasks (i.e., sota agents) and establishes a collaboration between these agents. however, this performance comes at a cost. the inference time is higher because it needs to run more than one model (at least one expert agent and one answer aggregator). therefore, we extend ukp-square to add support to the three main approaches for multi-agent systems, which we refer to as meta-skills on the platform: i) skill selection ( §4.1), ii) early-fusion of skills ( §4.2), and iii) late-fusion of skills ( §4. 3 ). an overview of the different architectures is illustrated in figure 1 . ## conclusions and discussions in this work, we have extended ukp-square to support multi-agent models. in particular, we deployed a routing system, tweac @xcite , a method to combine adapter weights, made @xcite , and a model that combines the prediction of multiple skills, metaqa @xcite . we have conducted experiments on these three models and unifiedqa @xcite , a multi-dataset system, to analyze the trade-off between the performance, efficiency, and flexibility of these systems. we showed that in scenarios where new domains or expertise are often needed, metaqa provides the best tradeoff since its performance is close to the best model, it is compatible with any qa format, cheap to train, and its inference runtime is close to tweac and made using the parallel engine provided by ukp-square. however, when simple deployment is needed or the model is not expected to be updated, made and unifiedqa might be more appropriate. this update of ukp-square is of utmost importance due to the current speed of development of qa models that creates thousands of models per year. our platform eases the deployment, running, comparison, and analysis of qa skills. with this update, we also facilitated the aggregation of these skills into multi-skills simplifying research on multi-agent systems. we leave as future work the comparison of these modular systems with prompting-based qa in large language models @xcite .