UKP - SQUARE : An Online Platform for Question Answering Research (2022)

Abstract: Recent advances in NLP and information retrieval have given rise to a diverse set of question answering tasks that are of different formats (e.g., extractive, abstractive), require different model architectures (e.g., generative, discriminative), and setups (e.g., with or without retrieval). Despite having a large number of powerful, specialized QA pipelines (which we refer to as Skills) that consider a single domain, model or setup, there exists no framework where users can easily explore and compare such pipelines and can extend them according to their needs. To address this issue, we present UKP-SQuARE, an extensible online QA platform for researchers which allows users to query and analyze a large collection of modern Skills via a user-friendly web interface and integrated behavioural tests. In addition, QA researchers can develop, manage, and share their custom Skills using our microservices that support a wide range of models (Transformers, Adapters, ONNX), datastores and retrieval techniques (e.g., sparse and dense). UKP-SQuARE is available on https://square.ukp-lab.de

## introduction researchers in nlp have devoted significant resources to creating more powerful machine learning models for question answering (qa), and collecting high-quality qa datasets. combined with the recent breakthroughs by large pretrained language models, we have witnessed rapid progress in the field across many different kinds of qa tasks @xcite . the great variety in qa tasks has led to specialized, domain-specific models trained on a single qa format such as multiple choice @xcite (i.e., selecting the best answer out of multiple options), extractive @xcite 1 the code is available on https://github.com/ ukp-square/square-core (i.e., finding the answer span in a context) and abstractive @xcite , generating an answer that is not a contiguous span in the context). the format may influence the model architecture (e.g., discriminative objective for multiple choice, generative objective for abstractive). additionally, systems vary with how the context is provided. it can be given by the user, or retrieved from a datastore which is commonly referred to as open-domain or retriever-reader setup @xcite . the retrieval mechanism can also be chosen from a set of sparse (e.g., bm25, @xcite or dense (e.g., dpr, @xcite techniques. the speed of progress in the field makes it essential for researchers to explore, compare, and combine these different qa components as quickly as possible to identify the strengths and weaknesses user interface explainability checklist datastores wikipedia bm25 dense bing web docs. bm25 dense pubmed bm25 dense wikipedia bm25 dense dense models span extraction adapter / hf / onnx query embedder sbert / adapter / hf / onnx text generation adapter / hf / onnx text classification adapter / hf / onnx skills open-domain extractive qa machine reading comprehension … 👉 user query 🔍 : when was tu darmstadt established? ... … … it was founded in 1877 … 3) query embedding [ 0.211 0.116, …, 0.202, 0.141 ] 2) document retrieval 4) answer extraction ... … … it was founded in 1877 … 6) explanations 5) return results: … it was founded in 1877 … 1) input query of the current state of the art. even though there exists a number of powerful qa systems @xcite ) and frameworks such as haystack, foot_0 those approaches focus only on one component (e.g., retrieval, qa format, domain), hence do not allow plug-and-play of different datastores, domains, model architectures or retrieval techniques. this considerably limits their applicability and reusability across the diverse, rapidly progressing area of qa research, making it infeasible for researchers to quickly integrate novel models and qa pipelines. to address this gap, we introduce ukp-square, a flexible and extensible qa platform to enable users to easily implement, manage and share their custom qa pipelines, which we call skills, using our microservices. as shown in fig. 1 , ukp-square also allows users to query and compare different skills via an easy-to-use user interface and systematically analyze their strengths and weaknesses through integrated behavioural tests. foot_1 ## ukp-square the system is implemented as a modern microservice architecture using docker containers. 4 the major components are skills, datastores, models, explainability and the user interface. the process flow across the components is illustrated in fig. 2 on an open-domain, extractive qa skill. the central component of the system is the skill that specifies how a user query is processed (e.g., which qa type, retrieval mechanism, model or adapter to be used in which order). the skill leverages the other services for query execution. datastores hold multiple collections of documents with sparse indices, e.g., bm25 @xcite and dense indices, e.g., dpr @xcite , allowing fast and efficient retrieval of background knowledge in an extensible way. the model service hosts numerous models, combined with adapters @xcite , to support a wide range of tasks such as text embedding (for queries in opendomain qa), sequence and token classification (for multiple-choice and extractive qa) and sequenceto-sequence generation (for abstractive qa). the explainability component performs behavioural tests on the deployed skills for better understanding of the models. details of each service are provided in the following sections. furthermore, while we host ukp-square on our infrastructure and make it available for the community, we also provide the option to set up the system locally. additionally, the datastores and models services are exposed via an api foot_3 . ## user study we evaluate the usability of our system by conducting a pilot attitudinal user study with five participants. we recruited graduate students, our main target user group, and instructed them to compare and analyze several skills. we provided them with a list of predefined questions to input into the system to help them use it. after the students used the system we asked them several questions to discover whether they understood every element of the interface effortlessly (i.e., the input and the output of the skills, the list of behavioral cards of the skills, and their specific contents). all users understood the input and output of the skills and stated that the interface allows them to compare the skills effortlessly. they also stated that the behavioral cards of the explainability component are useful to analyze the strong and weak points of the models and could help develop new skills. however, most of them could not understand them in a glimpse. hence, we will improve the presentation of these cards in a future update. appendix c provides the list of questions and responses. to finish the study, we employed the system usability scale (sus) questionnaire @xcite to quantitatively assess the global usability of the system. the average score is 70 out of 100, which refers to a "good usability" (uiux-trend, 2021). ## conclusion and future work we introduce the ukp-square platform that enables researchers and developers to study and compare qa pipelines, i.e., skills, that comprises a selection of datastores, retrieval mechanisms and reader models. the platform enables querying ex-isting public skills, as well as implementing custom ones using ukp-square's microservices and utility functions that support a large collection of model types and datastores. furthermore, users can simultaneously query multiple skills, and analyze them through integrated behavioural tests. our architecture is scalable and flexible to incorporate most of the latest developments in the qa domain. future versions will include automated deployment of custom models and datastores, automated skill selection by incorporating previous works @xcite and increasing the number of supported datastores (e.g., wikidata, @xcite . we also plan to incorporate specialized models (e.g., using graph encoders, ribeiro et al., 2021), structured reasoning approaches @xcite and interpretability techniques such as saliency maps @xcite .