  Question answering systems should help users to access knowledge on a broad
range of topics and to answer a wide array of different questions. Most systems
fall short of this expectation as they are only specialized in one particular
setting, e.g., answering factual questions with Wikipedia data. To overcome
this limitation, we propose composing multiple QA agents within a meta-QA
system. We argue that there exist a wide range of specialized QA agents in
literature. Thus, we address the central research question of how to
effectively and efficiently identify suitable QA agents for any given question.
We study both supervised and unsupervised approaches to address this challenge,
showing that TWEAC - Transformer with Extendable Agent Classifiers - achieves
the best performance overall with 94% accuracy. We provide extensive insights
on the scalability of TWEAC, demonstrating that it scales robustly to over 100
QA agents with each providing just 1000 examples of questions they can answer.
