QUADR o: Dataset and Models for QU estion-Answer Database Retrieval (2023)

Abstract: An effective approach to design automated Question Answering (QA) systems is to efficiently retrieve answers from pre-computed databases containing question/answer pairs. One of the main challenges to this design is the lack of training/testing data. Existing resources are limited in size and topics and either do not consider answers (question-question similarity only) or their quality in the annotation process. To fill this gap, we introduce a novel open-domain annotated resource to train and evaluate models for this task. The resource consists of 15,211 input questions. Each question is paired with 30 similar question/answer pairs, resulting in a total of 443,000 annotated examples. The binary label associated with each pair indicates the relevance with respect to the input question. Furthermore, we report extensive experimentation to test the quality and properties of our resource with respect to various key aspects of QA systems, including answer relevance, training strategies, and models input configuration.

## introduction recently, two main qa paradigms have gained more and more attention from both the industrial and research community: open-and closed-book. the former paradigm, also known as retrieve-andread refers to a set of models and techniques that, given an input question, follow a two-step process. first, they retrieve relevant content from a large corpus of documents, e.g. from the web. second, they extract the answer from the retrieved content @xcite @xcite . the latter approach, closed-book, is relatively new and it consists of models that rely on knowledge acquired during training, and they generate an answer without accessing external corpora. typically, these techniques are based on seq2seq models such as t5 @xcite . although * work done as an intern at amazon alexa ai these approaches have recently shown impressive performance, their execution places a substantial strain on system resources, making their adoption unpractical for industrial applications @xcite . a third approach that started to become popular in the last years consists in retrieving the answer from a database of pregenerated question/answer pairs rather than leveraging knowledge stored in the models' parameters or large corpora. these systems typically employ 3 components: (i) a database of questions and their answers, (ii) a retrieval model to query the database and to retrieve relevant content for an input question, and (iii) a ranking model to select the final answer. recent literature showed various benefits of these approaches, here named database qa (dbqa), including fast inference and the ability to introduce new knowledge by simply adding new pairs without re-training models. one of the main issues with dbqa approaches is the lack of large training data to support the development of retrieval and ranking models. only limited resources exist in the literature and often models are indeed trained on general tasks, including paraphrasing or semantic similarity @xcite . to fill this gap, we introduce a novel large annotated dataset to train and evaluate models for question ranking and dbqa. our resource consists of 15,211 open-domain input questions. for each input question, we annotated 30 relevant q/a pairs and we marked them as semantically equivalent or not (binary label) with respect to the input. in total, the dataset comprises ≈ 443, 000 annotated examples. we selected q/a pairs using a dbqa model trained on publicly available data. this allowed us to easily integrate labeled hard negatives, making the resource significantly complex and challenging. differently from existing resources mainly based on question-question pairs, our dataset is annotated both with respect to question-question similarity and question-answer correctness. to the best of our knowledge, this resource is the largest annotated dataset for training dbqa models. we performed extensive experiments, which provide baselines and a comprehensive analysis of distinctive aspects, weaknesses, and benefits of the dataset. as second contribution of this paper, we use the annotated resource to train models for dbqa. then, we build a standard dbqa pipeline, named quadro (question answer database retrieval) based on these models and we test it in various open domain qa settings. our experiments aim to address a critical gap in the existing literature by providing consolidated evidence regarding some key aspects of dbqa end-to-end pipelines. for instance, despite the existing evidence supporting the contribution of the answer to the retrieval task @xcite , its utilization remains poorly explored in dbqa applications @xcite . our annotated dataset and our models will be available to the research community foot_0 . more recently, pretrained transformers set the state of the art for sts and dqd @xcite . @xcite proposed tbert, a transformer model that takes the concatenation of the two questions as input, providing a joint and contextualized representation of the pair. database qa is a known paradigm that typically relies on (i) a curated database (or collection) of questions and their answers, (ii) a retrieval model to query the database (e.g. bm25 or dpr) for finding an equivalent question, and (iii) an optional ranking component. early works on dbqa for forum or faq @xcite @xcite have pointed out that when answers are available in the database together with the questions, the resulting system can be very accurate. the main challenge to their general usage is the typical specificity of the db, also associated with a limited availability of q/a pairs. @xcite introduced wekos, a system able to identify semantically equivalent questions from a faq database. the model, based on k-means clustering, word-embeddings (cbow), and heuristics, was tested on a dataset based on yahoo answer, showing impressive performance compared to the existing systems. after the rise of transformer models, @xcite proposed a new ensemble system that combines bm25 with bert @xcite for retrieval and ranking. the same authors also explored systems based on a gpt2 model to generate question and/or answer paraphrases to fine-tune the bert models on lowresource faq datasets with promising results. similarly, @xcite proposed a method based on bert and tsubaki, an efficient retrieval architecture based on bm25 @xcite , to retrieve from a database of faq the most similar questions looking at query-question similarity. however, most of these works and practical applications were confined to specific domains, including domain-specific faq or community qa. recently, effective open-domain dbqa pipelines have been shown. @xcite assembled a database of 65m q/a pairs automatically generated from the web. then, they employed a neural retrieval and ranker based on transformer models to query the database. the authors compared this end-to-end dbqa pipeline, named repaq, against various closedand open-book qa strategies, showing interesting performance in terms of efficiency and competitive accuracy. however, there are some limitations of that system as (i) models are not trained on annotated data for question ranking due to lack of available resources and (ii) the database is noisy (estimated 18% incorrect q/a pairs). as an extension of this approach, @xcite proposed a 2 steps retrieval based on bm25 and dpr executed in sequence to improve the efficiency. however, most of the technical details are missing, including the database used and training mechanisms. it is worth noticing that these end-to-end systems rely solely on models trained for question similarity, without providing empirical evidence of the advantages gained from incorporating answers to enhance accuracy. ## database qa architecture inspired by previous work @xcite , we consider a retrievalreranking dbqa architecture consisting of a largescale database of questions and correct answers, an efficient search engine to query the database, and an answer selector (ranker). in the remainder of the paper, we call this dbqa system quadro (question answer database retrieval). search engine this is based on recent findings in neural retrieval, including dense passage retrieval @xcite . it consists of a siamese bi-encoder transformer (reimers and gurevych, 2019) (also known as sentence-transformer) network. the first branch encodes the input question t as [cls] t [eos], whereas the second branch encodes question/answer pairs (q i , a i ) from the database as [cls] q i [sep] a i @xcite . the cosine similarity between the representations extracted from the two branches expresses the level of semantic similarity between the question and the q/a pair. let δ : σ * → r d be the transformer function which maps a text @xmath0, an answer a @xmath1, or a pair (q i , a i ) ∈ q × a) into a d-dimensional embedding. the final score assigned to a target question @xmath2, a i ) is δ(t) ⊤ δ(q i ,a i ) ∥δ(t)∥∥δ(q i ,a i )∥ , where ∥ • ∥ denotes the 2-norm of a vector. when the user asks a new question, the biencoder computes the cosine distance between the question embedding and all q/a pairs and returns the most similar k pairs. answer selector after the retrieval stage, an answer selector (or reranker) model re-ranks the pairs returned by the search engine and selects the final answer to be served to the user. formally, let @xmath3, a i )} k @xmath4. the answer selector r : q × @xmath5, r(t, q i , a i ), and returns the answer associated with highest ranked pair, i.e., arg max @xmath6...k r(t, q i , a i ). inspired by the contextual sentence selector framework @xcite , which is the state of the art for open-domain answer selection, we use a transformer model to encode and rank triplets. the input of the transformer is encoded as [cls] t [sep] a i [sep] q i @xcite . note that this allows to jointly modeling the semantic dependencies between the two questions, e.g., their similarity, and the rele-vance of the answer to both questions. the database the database employed in this work consists of questions and their answers collected from various heterogeneous public highquality annotated open-domain qa datasets, including: gooaq, wqa @xcite , wikianswer @xcite , covidqa @xcite , and hotpotqa @xcite . we extracted questions and the associated correct answer text span from these datasets, and ingested these q/a pairs into our database. beyond annotated resources, we enhanced our database with various sets of artificially generated q/a pairs. first, we considered questions from quo-raqp. differently from the other datasets above, quoraqp is designed for question duplication detection task and not qa. thus it simply consists of question pairs. some answers are available for a small fraction of questions, which were selected through a heuristic approach based on the rank of users' content in quora threads @xcite . to expand the quoraqp, we collected the missing answers using a similar approach described by @xcite : given an input question, we queried a 2020 commoncrawl snapshot 2 using bm25 and we selected the 200 most relevant documents. then, documents are split into sentences and a state-of-the-art sentence selector (lauriola and moschitti, 2021) is used to select the topranked passage as the answer. in addition, the score returned by the passage reranker model can be effectively used to measure the likelihood of the answer being correct. we applied a threshold to this score and accepted only the top 10% of q/a pairs. this guarantees a higher level of answer quality in our database. we manually labeled 200 randomly selected answers from this unsupervised set as quality assessment, observing an accuracy of 93.0%. finally, we also ingested q/a pairs from eli5 @xcite . this dataset consists of questions collected from three subreddits for which the answers have been ranked by the users' up-votes in the thread. although this heuristic removed part of the noise of the dataset, to ensure the maximum quality we keep only the 50% of the q/a pairs with the highest sentence selector score. after a manual annotation, we estimate the 84.3% of accuracy. our final db contains ≈ 6.3 millions of english questions and their correct answers pairs. further details and statistics are reported in ## annotated dataset as section 2 highlights, existing resources are not suited to train models for dbqa. in this section, we describe the annotation workflow, emphasizing the main features and key aspects. ## experiments we ran various distinct sets of experiments to assess the quality and challenges of our dataset and to show the empirical performance of state-of-the-art models in dbqa tasks. first, we analyze the retrieval and ranking tasks thanks to our new dataset. then, we use this resource to fine-tune and evaluate retrieval and ranker models. finally, we compare our approach against strong web-based opendomain qa systems. as we previously mentioned, our experiments aim to empirically clarify and report some design choices of dbqa systems including, for instance, the modeling of the answer. ## conclusions end-to-end qa pipelines that find the answers in a precompiled database of question/answer pairs (dbqa) have become popular in recent years due to their efficiency, accuracy, and other benefits. one of the main issues of these approaches is the lack of large training/test data to develop competitive models. existing resources are limited in dimension, scope, or they do not consider the relevance of the answer. in order to fill this gap, we introduce a novel annotated resource to train models for dbqa. our dataset comprises 443,000 examples, making it one of the largest annotated resources for this task. another novelty is the annotation mechanism that considers both, the similarity of the question with respect to the input and the relevance of the answer. furthermore, our experiments report various key aspects of dbqa systems often neglected in the literature, including the contribution of the answer in the retrieval and ranking tasks and the need for high-quality training data. we believe that our data and models, made available to the research community, will enable interesting future research in optimizing models for dbqa. for example, how to improve the selection of correct q/a pairs in the top k pairs.